Biddle me this? (or Flunkout Nation)

While I suspect few people have read or seen this post by RiShawn Biddle of Dropout Nation, I felt that it was worth mentioning because it presents such egregiously flawed logic coupled with flat-out factually incorrect and unsubstantiated claims. Sadly, this is what we have come to all too often in the current education reform debate. And this isn’t really about RiShawn Biddle as an individual or his blog and tweets, but rather about the propensity to argue important and complex issues in such crude terms and with so little knowledge or understanding of context and history.

In a recent post, Biddle argues that the NAACP is and has been heading up a misguided public policy agenda on behalf of black America (my characterization – perhaps not right on target). Biddle argues that Jealous should a) admit that arguments for more funding and more equal funding are wrong and have proven to fail and b) that instead, Jealous should embrace charter schools as a solution.

Now, this is a strange dichotomy to begin with – either fair and adequate funding or charter schools. It seems from a charter advocacy approach that one would also want fair and adequate funding including substantial funding targeted to high need areas. That is, both, not either/or.

That logical point aside, Biddle then goes on to make his bold points to Jealous with the most absurd claims I’ve read, in well, about a week.

Here’s how Biddle explains the failures of legal challenges over school funding:

The NAACP has taken the wrong approach on school reform for far too long. The continuing dropout factory status of Newark, Kansas City, Mo., and other cities that have benefited from funding equity suits is clear evidence that this approach doesn’t spur any kind of reform.

I’ll set aside the equally absurd claims about integration that follow in the next sentence. But, let’s take a quick look at this claim.

First of all, Kansas City, Mo. never really benefited from a case over school funding. Rather, Kansas City, Mo. – this supposed poster child for failed school funding reform – saw a short term boost in funding while under court ordered desegregation. Funding litigation concurrent (1993) with the desegregation litigation had negligible effect on KCMSD funding. Later funding challenges in Missouri were found in favor of the state, producing no benefit to Kansas City, Missouri. In fact, from about 1995 to present, KCMSD funding has generally slid backwards.

Preston Green and I document the disconnect between desegregation litigation in Kansas City and claims of school funding failures in this article: Urban Legends, Desegregation and School Finance: Did Kansas City Really Prove That Money Doesn’t Matter? (which appeared in the Michigan Journal of Race and Law). Among other things, we note:

Critics cite the statistics the KCMSD spent more than $11,000 per pupil and that $2 billion were spent on the desegregation plan as evidence of exorbitant spending. When taken out of context, these numbers appear huge. However, our analysis reveals that the KCMSD was a very high spending district for no more than five years, or the time in which one cohort of children is able to progress through five grade levels in the district. Further, when adjusted for student needs, the KCMSD’s funding dropped below the metropolitan area average by 1998. This is hardly enough time to erase the generational poverty of the KCMSD or alter the residential structure and demographics of a school district that had been designed to be racially segregated until the 1960s.

Biddle also points to Newark, NJ as providing evidence of the failures of school finance reforms. Yet, New Jersey is among those states we discuss here as having some (albeit limited due to data quality) evidentiary basis for the positive effects of state school finance reforms, including court ordered reforms. In School Finance and Courts: Does Reform Matter, and How Can We Tell?, Kevin Welner and I discuss the many flawed claims about the dreadful failures of attempts to improve equity and adequacy of school funding. We also point out how the Kansas City case does not even fit into this category. Regarding the general, popular claims of the failures of funding reforms, Kevin Welner and I review the basis for those claims and conclude:

We conclude that there is arbitrariness in how research in this area appears to have shaped the perceptions and discourse of policymakers and the public. Methodological complexities and design problems plague finance impact studies. Advocacy research that has received considerable attention in the press and elsewhere has taken shortcuts toward desired conclusions, and this is troubling.

We also review more rigorous peer-reviewed studies and find, on balance, that those studies show positive effects of school finance reforms, both in terms of improving equity in student outcomes and in terms of improving the overall level of student outcomes.

Higher quality research, in contrast, shows that states that implemented significant reforms to the level and/or distribution of funding tend to have significant gains in student outcomes. Moreover, we stress the importance of the specific nature of any given reform: positive outcomes are likely to arise only if the reform is both significant and sustained.

And now for the truly ironic part of Biddle’s claim. So, Biddle’s argument to Jealous is that Jealous should drop all this funding equity and integration crap from NAACP’s past, and focus on charters – expanding access to charters. After all, if Kansas City, Missouri had not wasted all that time arguing in court over money and chasing more equitable funding and instead had spent its time pursuing an aggressive strategy of increasing numbers of charter schools, kids in Kansas City – especially poor, minority kids – would have much better educational opportunities!!!!

Biddle argues (regarding NAACP):

It must embrace the charter school movement: After all, charters have been the leading source of improving access to high quality education for urban black and Latino communities, who would otherwise be forced to attend the dropout factories in their neighborhoods. The success of charter school operators such as KIPP and Uncommon Schools — all of which educate mostly-minority students — can be replicated throughout the nation.

I wouldn’t have even written this response (to a post not really worthy of response) had it not been for the fact that I addressed this very topic the other day. As funding for KCMSD was receding as the district moved toward unitary status, what did happen in KCMSD? The massive expansion of charter schools! Here’s what I wrote the other day, upon release of a very interesting report from Kauffman Foundation regarding educational opportunity in Kansas City and the role of charter schools:

Kansas City is #4 on charter market share, according to the National Alliance report, and rose to that position much earlier in the charter proliferation era than other cities. As a result, by reformy logic, Kansas City should be a hotbed for educational opportunity for school-aged children – after years of previously throwing money down the drain in the Kansas City Missouri Public School District (many of these claims actually being Urban Legend).

In Kansas City, the reality of charter expansion has clashed substantially with the reformy ideology. Arthur Benson in a recent Kansas City Star Op Ed, noted:

Charters have subtle means for selecting or de-selecting students to fit their school’s model. The Kansas City School District keeps its doors open to non-English speakers and all those kids sent back from the charter schools. In spite of those hurdles, Kansas City district schools across the board out-perform charter schools. That is not saying much. We have until recently failed 80 percent of our kids, but most charters fail more.

I was initially curious about Benson’s (a district board member and attorney) claims that charters have done so poorly in Kansas City. Could it really be that the massive expansion of charter schools in Kansas City has done little to improve and may have aided in the erosion of high quality educational opportunities for Kansas City children?

The recent Kauffman Foundation report draws some similar conclusions, and Kauffman Foundation has generally been an advocate for charter schools. The report classifies district and charter schools into groups by performance, with level 4 being the lowest, and level 1 being the only acceptable group.

  • Level I- A school that met or exceeded the state standard on the MAP Communication Arts and Mathematics exams in 2008-2009.
  • Level II- A school that scored between 75 and 99 percent of the state standard on the MAP Communication Arts and Mathematics exams in 2008-2009.
  • Level III– A school that scored between 50 and 74 percent of the state standard on the MAP Communication Arts and Mathematics exams in 2008-2009.
  • Level IV– A school that scored below 50 percent of the state standard on the MAP Communication

Among other things, the report found that charter operators had avoided opening schools in the neediest neighborhoods. Rather, they set up shop in lower need neighborhoods, potentially exacerbating disparities in opportunities across the city’s zip codes. The report recommended:

A strategy for charter school growth should be developed by Kansas City education leaders. Charter schools should only be approved by DESE if they can demonstrate how they intend to fill a geographic need or a specific void in the communities they intend to serve.

Regarding charter performance more generally, the report noted:

In many communities charter schools are a model that increases students’ access to better public schools, but the majority of charter school students (5,490 or 64.7 percent) are in a Level IV school. Many of Kansas City’s charters have existed for 10 years and are still not able to reach even half of state standard.

Now, I’m not sure I accept their premise that in many communities this actually works – and that it just went awry for some strange reason in Kansas City. That said, the reality in Kansas City, by the authors own acknowledgment is in sharp contrast with the reality the authors believe exists in other cities.

One implication (not tested directly) of this report is that the massive charter school expansion that occurred in Kansas City may have done little or nothing to improve the overall availability or distribution of educational opportunities for children in that city and may have actually made things worse.

So, Mr. Biddle, I urge you to do a little reading. Check a few facts and consider your arguments carefully. Your current arguments FLUNK at even the most basic level.

More importantly, others who come across such bombastic claims like those argued by Biddle should scrutinize those claims carefully. Heck, I hope you scrutinize the stuff in my own posts carefully too. I try to shoot for a reasonably high level of rigor and factual accuracy in these posts and do what I can to cite my claims to respectable sources. Biddle’s blog to Benjamin Jealous is, to me, an example of the worst form of ill-conceived, factually incorrect, contorted reform logic out there. Sadly, there’s way too much of it.

READINGS

Baker, B.D., Welner, K. (2011) School Finance and Courts: Does Reform Matter, and How Can We Tell? Teachers College Record 113 (11) p. –

Green, P.C., Baker, B.D. (2006) Urban Legends, Desegregation and School Finance: Did Kansas City really prove that money doesn’t matter? Michigan Journal of Race and Law 12 (1) 57-105

When schools have money…

When schools and school districts have more money and spend more money, what do they spend it on?

We are told these days to believe that everything we thought about the virtues of small class size back in the 1990s was misguided. That improving teacher quality trumps reducing class size any day when it comes to efficiently improving student outcomes. We are told to believe that teacher quality can be improved at nominal cost, whereas achieving similar gains via class size reduction would be absurdly inefficient and very costly. Yet to date, we have little evidence that we can actually achieve the same measured outcome gains achieved by reducing class size in the 1990s by instead improving teacher quality… and that this can easily be done at lower cost. In fact, many go so far as to argue that we can take the same average teacher wage, and instead of paying older teachers more and younger teachers less, we just have to pay better teachers more and worse ones less – and that somehow this will lead to a new interest in teaching among our best and brightest college graduates. I struggle with the reasoning here, and certainly have not seen the evidence.

I am particularly skeptical that dramatically reducing the predictability and stability of career earnings, while not altering dramatically the average level of compensation can result in any positive changes to teacher quality. This is especially true if higher teacher wages are tied to extremely noisy measures of teacher performance – making it difficult for a teacher to even control, no less predict his/her career earnings.

We do know from many older studies that improving wages can improve the teacher workforce:

  • Murnane and Olson (1989) find that salaries affect the decision to enter teaching and the duration of the teaching career.[1]
  • Figlio (1997, 2002) and Ferguson (1991) find that higher salaries are associated with better qualified teachers[2]
  • Loeb and Page (1998, 2000) find that raising teacher wages by ten percent reduces high school dropout rates by between three and six percent and increases college enrollment rates by two percent.[3]

We also know that imposing strict spending limits on public schooling, thereby limiting the ability of public schools to pay competitive salaries can harm teacher quality over time:

  • Figlio and Rueben (2001) explain: “Using data from the National Center for Education Statistics we find that tax limits systematically reduce the average quality of education majors, as well as new public school teachers in states that have passed these limits.”[4]

Interest in teacher quality over class size reduction has grown so strong that some are beginning to make the leap that we should simply increase class size to 30 or even 35 students per class in order to pay enough to get really good teachers. After all, who can argue with the logic that a good teacher with 35 kids is better than a crappy one with 20 kids.  Of course, this assumes falsely that every class of 35 would be taught by a better teacher, on average, than those teaching the classes of 20, because every teacher currently teaching the smaller classes is crappy. That said, we do have pretty consistent evidence that salary increases could increase teaching quality.

However, we also have at least some evidence that teacher quality and class size interact.  We may find that we are fighting a losing battle trying to recruit high quality teachers to teach classes of 35 kids even at the higher salary. This may especially be the case in schools and districts where large classes are particularly difficult to manage. Class size is a working condition and more desirable working conditions can reduce the need for paying higher salaries – another trade-off for which we have no good dollar to dollar estimates.

This brings me to my somewhat related data query for the day. When schools have money to spend, what do they spend it on? When looking at high spending suburban school districts, or looking at private independent schools, which I have referred to elsewhere as “luxury” schooling, what are their defining attributes?

Arguably, the defining attributes of luxury schooling simply reflect the demands of luxury schooling consumers – residents of high spending affluent suburban communities and parents who send their children to private independent schools. Go to nearly any private school web site – or web site on “why you should choose a private school,” and you will find one item at the top of the list nearly every time – Small Class Size- or Individual Attention (and alternative angle on what? small class size!). But this is irrational right? Why should affluent suburban consumers or private school parents prefer something that simply drives up the price and with diminishing marginal returns? Whatever the reasons, they do, and arguably lower pupil to teacher ratios and smaller class sizes are a, if not the defining feature of “luxury” schooling.

First, here are the per pupil spending levels in select labor markets, for private schools by type and for public schools in the same labor market.

FIGURE 1 : Per Pupil Spending of Private and Public Schools by Labor Market

Private independent schools in particular, systematically outspend public schools in the same labor market by about 2/1!

And this graph shows the pupil to teacher ratios for public schools, all private schools, private Catholic schools and private independent schools. Private independent schools spend double what public schools spend, and leverage most of that money to provide pupil to teacher ratios that are approximately half those of the public schools (teacher salaries are similar to slightly lower than public school salaries).

FIGURE 2: Pupil to Teacher Ratios of Private and Public Schools

Now, this graph shows the per pupil state and local revenues of public school districts in the NY metropolitan area, by district poverty rates. In New York State, as we show at schoolfundingfairness.org, higher poverty districts have systematically fewer resources than their lower poverty, often very affluent suburban neighbors. This graph validates that pattern.

FIGURE 3: Per pupil Revenues of New York Metro Area Districts (in NY State) by Poverty

Now here are the elementary class sizes by district spending group. Note that as spending per pupil increases, class sizes systematically decrease.

FIGURE 4: Spending and Elementary Class Size

The same pattern holds for middle and secondary class sizes.

FIGURE 5: Spending and Middle/Secondary Class Size

Note that at least some of the smaller class size at the middle/secondary level in the highest spending public school districts is a function of providing a diverse set of specialized elective courses, advanced placement classes, multiple languages and so on. The same is true for private independent schools. These are opportunities that many lower spending and/or higher poverty districts in many states go without.

Yes, consumers of luxury schooling seem to have a pretty strong preference for small classes, despite modern wisdom that class size is clearly second fiddle to teaching quality. Imagine the teacher salaries one could pay by moving pupil to teacher ratios in independent schools from 8/1 up to the public school average of 16/1. Imagine the salaries that could be paid in affluent Westchester County and Long Island school districts by increasing class sizes from 16 or 18 up to 35? (see this post on just how high these salaries already are!)

For some reason these private schools and affluent public school districts – more specifically those who support these schools – exhibit a strong preference for small class size even when given wide latitude to cho0se differently. Perhaps they are on to something?

References

[1] Richard J. Murnane and Randall Olsen (1989) The effects of salaries and opportunity costs on length of state in teaching. Evidence from Michigan. Review of Economics and Statistics 71 (2) 347-352
[2] David N. Figlio (1997) Teacher Salaries and Teacher Quality. Economics Letters 55 267-271. David N. Figlio (2002) Can Public Schools Buy Better-Qualified Teachers?” Industrial and Labor Relations Review 55, 686-699.  Ronald Ferguson (1991) Paying for Public Education: New Evidence on How and Why Money Matters. Harvard Journal on Legislation. 28 (2) 465-498.

[3] Susanna Loeb and Marianne Page (2000) Examining the link between teacher wages and student outcomes: the importance of alternative labor market opportunities and non-pecuniary variation. Review of Economics and Statistics 82, 393-408. Susanna Loeb and Marianne Page (19980 Examining the link between wages and quality in the teacher workforce. Department of Economics, University of California, Davis

[4] David N. Figlio and Kim S. Rueben (2001) Tax limits and the qualifications of new teachers Journal of Public Economics Volume 80, Issue 1, April 2001, Pages 49-71

Spotlight on Ideologies at the Extreme in New Jersey

About a week ago, I characterized “reformy” ideologies in a way that I myself even thought was extreme – so much so, that later on I added caveats to my description to say that this really isn’t what most “reformers” are advocating, but rather “reform” at the extremes.

  • Reformy Ideology #1: Teacher quality is the one single factor that has the greatest effect on a child’s life chances. Get a bad teacher or two in a row, and you’re screwed for life. The “best possible” way to measure teacher quality is by estimating the teacher’s influence on student test scores (value-added). Hiring, retention and dismissal decisions must, that is, MUST be based primarily on this information. This information may be supplemented, but value-added must play the dominant single role.
  • Reformy Ideology #2: Charter schools are the answer to most of the problems of poor urban school districts. Take any poor, failing urban school district, close those dreadfully failing schools and replace them as quickly as possible with charter schools and children in the urban core will have greatly expanded high-quality educational opportunities.

In that same post, I raised some questions about both ideologies, as I’ve done in many previous posts. In short, the “solution” part of reform ideology #1 is deeply problematic for a multitude of reasons and I urge you to read my entire thread on value-added models for assessing teacher effectiveness: https://schoolfinance101.wordpress.com/category/race-to-the-top/value-added-teacher-evaluation/

In addition to discussing the multitude of technical issues with value-added measures specifically, I also discuss extensively the potential labor market consequences for high poverty schools.

As I noted on my post the other day the problem with Reform Ideology #2 is not so much about specific charter schools or specific models and/or whether some work well or don’t, but rather about the idea that massive charter expansion is a panacea for the problems of poor urban districts. I’ve written much about his topic on this blog in the past: https://schoolfinance101.wordpress.com/category/charter-schools/

Who in their right mind would really argue that the solutions to all of our problems – or at least to urban education (code for poor and minority education) problems – is  as simple as charter school expansion, merit pay and tying teacher evaluation to test scores? Really, are there that many, or any, out there who still stick by this two-prong, unfounded, deeply problematic set of reform strategies?

Just when I thought I might have gone overboard and perhaps even been too unfair to the reform crowd, I read this editorial in New Jersey Spotlight today: http://www.njspotlight.com/stories/10/1024/1915/

Here are the conclusions of the editorial:

Here’s one way to get through the rational albeit provincial resistance from leaders of high-performing districts. Let’s just say, we have schools that are among “the very best in the nation.” But we also have schools that are among the nation’s worst. We’ve made this distinction for years, primarily through the State Supreme Court Abbott decisions, which mandate that we fund our poorest districts (recently revised to poorest students) at the same rate as our wealthiest. Why not take this acknowledgement of inequity to its logical conclusion and implement reform efforts — charter school expansion, school choice, higher compensation for great teachers, data-driven instruction — in our chronically failing districts?

Surely school leaders, legislators, New Jersey Education Association (NJEA) executives and the DOE can coalesce around charter school expansion in Pleasantville and Trenton; merit pay in Camden and Plainfield; or tying student growth to teacher evaluations in Newark and Asbury Park. While state-wide school reform will eventually come to New Jersey, our poorest students can’t wait. Targeting progressive educational strategies to failing schools may be politically distasteful, but it’s the only way to get those kids under that big white tent where they belong.

In short, the author is explaining in the first part of the editorial that many wealthy, successful New Jersey school districts haven’t supported aggressive statewide “reformy” strategies because they want no part in those strategies in their own districts. The same districts have been tentative about expanded choice for inter-district transfers. But, as this editorial argues, these districts should band together… should coalesce, to RAM DESTRUCTIVE, ILL-CONCEIVED POLICIES DOWN THE THROATS OF THEIR POOR URBAN NEIGHBORS. That’ll fix ’em! And without comparable adverse effects on their own districts!

I must say that this is about the most offensive call to arms I believe I’ve read in recent months. Yes, I’ve read some absurd arguments, like the argument that the “upper half of charters is better than average” or the argument that if current teacher evaluations are flawed, then the only answer is to replace them with student test scores (and other absurd false dichotomies).

The present NJ Spotlight argument begins with a deeply distorted, selective “factiness” about the failures of New Jersey’s urban districts (some of the nation’s worst! evidence?) and reasons for them (not enough charters, and no merit pay for teachers) and then jumps quickly to the most extreme and dreadfully oversimplified representation of the solutions (solutions, mind you, that may be far worse than the “disease”) to all of our – excuse me – their problems.

All when I thought that I might be getting too tough on the overly simplistic, bombastic and misguided logic of reform.

Teacher “effectiveness” ratings Freedom of Information Requests

Andy Rotherham over at Eduwonk posted an Irony Alert yesterday as many media outlets poised themselves to start “outing” ineffective teachers by posting publicly those teacher’s value-added effectiveness scores. Rotherham argued:

In light of this blow up about value-added in New York City, in a lot of places if the teachers unions would actually get serious about actually using value-add data as part of teacher evaluations it could be shielded from “Freedom of Information”requests that identify teachers, just as many aspects of personnel evaluations are.   They’re caught in their own mousetrap here.  My take on the larger issue from a few weeks ago and LA.

I thought…. hmmm… really? That doesn’t seem right. Is this just a clever argument intended to dupe teachers into getting those scores into their evaluations on some false assumption that the information would then be protected? Are these issues even transferable from state to state? Is the raw data used for generating the teacher effectiveness ratings actually considered part of the personnel file? I’m somewhat of an amateur on this school law stuff, but have enough background to start asking these questions when such arguments are tossed out there. So I did. I asked a handful of legal scholars in education policy, each of whom deals regularly with legal questions over personnel records under state law and with student record information.

Justin Bathon over at Ed Jurist has now posted his conversation starter for the legal community.

This is good stuff, and the very kind of conversation we should be having when such questions are raised. Ask the experts. Much of the argument hinges on when the raw data is translated into a measure that actually becomes part of the personnel file (at least with regard to the “shield” issue posed by Rotherham). Here’s Justin Bathon’s summary:

Anyway, summarizing, I think the raw data is generally going to be made publicly open following FOIA requests. I think New York City is currently correct in their assessment that no exemption exists under New York’s Freedom of Information Law. However, this is just my analysis after considering this issue for a single day and I want to caution against over reliance on my initial assumptions. A thorough analysis needs to be conducted of all 50 state policies, interpreting regulations, attorney general opinions, and previous case-law. Further, data experts such as Bruce must assist the analysis with a complete understanding of each state’s dataset and the possible links to both teachers and their evaluations within the datasets. Thus, there is still a lot of work left to be done.

This is a legal frontier (another one of those enabled by technology) that most legislatures would not have contemplated as possible in enacting their open records laws. Thus, it is a great topic for us to debate further to inform future policy actions on open records personnel evaluation exemptions.

Please, read the rest of his well thought out, albeit preliminary post.

Here are my follow-up comments (cross-posted at edjurist) on data/data structures and their link to teacher evaluations:

Here are some data scenarios:

A. The district has individual student test score data that are linkable to individual teachers but the district doesn’t use those data to generate any estimates of individual teacher “effectiveness,” has not adopted any statistical method for doing so and therefore does not include any such estimates as part of personnel records. The individual students’ identity can be masked but with matched ID over time and specific characteristics attached (race, low-income status)
B. The district has individual student test score data that are linkable to individual teachers just as above, and the district does a) have an adopted statistical model/method for generating teacher value added “effectiveness” scores, but uses those estimates only for district level evaluation/analysis and not for individual teacher evaluation.
C. The district has individual student test score data that are linkable to individual teachers as above, and a) the district has an adopted statistical method/model for generating teacher value-added “effectiveness” scores and has negotiated a contractual agreement with teachers (or is operating under a state policy framework) which requires inclusion of the “effectiveness” scores in the formal evaluation of the teacher.

Under option C above, sufficient technical documentation should be available such that “effectiveness” estimates could be checked/replicated/audited by an outside source.  That is, while there should be materials that provide sufficiently understandable explanations such that teachers can understand their own evaluations and extent to which their “effectiveness” ratings are, or are not under their own control, there should also be a detailed explanation of the exact variables used in the model, the scaling of those variables, etc. and the specification of the regression equation that is used to estimate teacher effects. There should be sufficient detail to replicate district generated teacher effectiveness scores.

That aside, a few different scenarios arise.

1. The LA Times scenario, as I understand it, falls under the first conditions above. The data existed in raw form. The district was not using those data for “effectiveness” rating. The LAT got the data and handed them over to Richard Buddin of RAND. Buddin then estimated the most reasonable regression equation he could with the available data and, for that matter, produced a sufficiently detailed technical report – such that anyone accessing the same data could replicate his findings. I suspect that individual student names were masked, but the students were clearly matched to identifiable teachers, and student data included specific identifiers of race, poverty, etc. and participation in programs such as gifted programs (indicator on child that he/she labeled as gifted). Not sure what, if any, issues are raised by detailed descriptive information on child level data. In this case, the data requested by LAT and handed over to Buddin were not linked to teacher evaluation by the district itself, in any way, as I understand it.

2. As I understood the recent NYC media flap, the city itself was looking to report/release the value-added ratings and the city itself also intends to use those “value added” ratings for personnel evaluation. It sounded to me that Charleston, SC was proposing roughly the same. Each teacher would have a nifty little report card showing his her “relative” effectiveness rating compared to other teachers. This effectiveness rating is essentially a “category” labeling a teacher as “better than average” or “worse than average.” These categories are derived from more specific “estimates” which come from a statistical model, which generates a coefficient for each teacher’s “effect” on the students who have passed through that teacher’s classroom (these coefficients having substantial uncertainty and embedded bias which I have discussed previously… but that’s not the point here). So, the effectiveness profile of the teacher is an aggregation of these “effects” into larger categories – but is nonetheless directly drawn from these effect estimates generate by the district itself for teacher evaluation purposes (even if subcontracted by the district to a statistician). I would expect that the specific estimate and the profile aggregation would be part of the teacher’s personnel record.

So, now that the city’s official release of effectiveness profiles is on hold, what if a local newspaper requested a) the raw student data linkable to teachers, with student names masked but with sufficient demographic detail on each student and with identifiable information on teachers, and b) the detailed technical documentation on the statistical model and specific variables used in that model?  The newspaper could then contract a competent statistician to generate his/her own estimates of teacher effectiveness using the same data used by the district and the same method. These would not be “official” effectiveness estimates, nor could the media outlet claim them to be. But they would be a best attempt at a replication. Heck, it might be more fun if they used a slightly different model, because the ratings might end up substantially different from the district’s own estimates.  Replicating or not, the districts own methods, and producing roughly the same or very different ratings for teachers, these estimates would still not be the official ones. Given the noise and variation in such estimates at the teacher level, it might actually be pretty hard to get estimates that correlate substantially with the district’s own estimates – and one would never know, because the district official effectiveness estimates for teachers would still be private.

I would assume under these circumstances, partly because the “official” personnel file estimates would remain unknown, and because it’s highly probable that the independent estimates produced by the media outlet – even if trying to replicate district estimates – might vary wildly from the district estimates – that the media outlet could get the data, estimate the model and report their results – their unofficial results. On the one hand, the media outlet could rely on the uncertainty of the estimates to justify that what they produce should not be considered “official” estimates. And on the other hand… in bold print in the paper… they could argue as the LA Times Jasons have … that these estimates are good and reliable estimates of actual teacher effectiveness!

The conversation continues over at EdJurist: http://www.edjurist.com/blog/value-added-evaluation-data-and-foia-state-versions-that-is.html?lastPage=true#comment10260419

Interesting follow-up point from Scott Bauries over at Ed Juris:

Thus, from the legal perspective, I am left with one question: if the data and conclusions are being used as reflected in option “c,” but the media only gets the conclusions and not the raw data, then does the law allow a teacher to protect his or her reputation from unfair damage due to the publishing of a conclusion based on a noisy equation?

This is a very complicated question, involving both defamation law and the First Amendment. For example, is a public school teacher a “public official” or “public figure” for First Amendment purposes, such that the standard for proving defamation per se is increased? If so, then is the relevant statistical analysis illustrating the noisy nature of the conclusion enough to show falsehood for the purposes of a defamation claim? I think probably not in both instances, but I don’t think this precise issue has ever come up.

When reformy ideologies clash…

(note: lots of ideas here that I wanted to start writing about… but not yet well organized or articulated. It will come, with time, I hope.)

Summary of Reformy Ideology

Bluntly stated, the two major components of education reform ideology are as follows:

  • Reformy Ideology #1: Teacher quality is the one single factor that has the greatest effect on a child’s life chances. Get a bad teacher or two in a row, and you’re screwed for life. The “best possible” way to measure teacher quality is by estimating the teacher’s influence on student test scores (value-added). Hiring, retention and dismissal decisions must, that is, MUST be based primarily on this information. This information may be supplemented, but value-added must play the dominant single role.
  • Reformy Ideology #2: Charter schools are the answer to most of the problems of poor urban school districts. Take any poor, failing urban school district, close its dreadfully failing schools and replace them as quickly as possible with charter schools and children in the urban core will have greatly expanded high-quality educational opportunities.

Now, let me do a bit of clarification here. These are the representations of reform ideology at the extremes – but these views are not uncommon in the “reform” community. Let me also clarify that item #2 above isn’t about the broader issue of charter schooling, the origins of charter schooling, purposes of individual charter schools or research on the effectiveness of specific charter school models. Item #2 above is specifically about the argument that large urban districts can and should be replaced with an open market of charter schools – that charter schools should not just be used to try out new and interesting ideas which may be replicated in other schools – charter or not – but rather that charters should become dominant providers of urban education.

In my framing of item #1 above, I do not by any means intend to discredit the importance of high quality teachers. I’m with the “reformers” on that idea. But, it is certainly an overstatement to attribute all of student gains and life chances to teacher quality alone. And, as I have discussed previously on many occasions on this blog, it is very problematic to assume that we presently have sufficient tools for measuring precisely and accurately the true effectiveness of any single teacher.

So, this brings me to recent completely unrelated events and media on education reform issues that raise some interesting points of conflict.

Part I: Ideology Clashing with… Ideology

The first example of the clash of reformy ideologies comes from upstate New York as the state begins the process of implementing the “reforms” that got that state Race to the Top funding. In short, charter school operators really don’t seem to want to be compelled to adopt the first prong of reformy ideology. What? Charters don’t want to be compelled to use student test scores as the primary, or even a major basis for personnel decisions? Blasphemy!

In recent weeks, in casual conversations and at symposia, I’ve actually heard a number of charter school operators raise serious questions about being compelled to adopt Reform Ideology #1 above. Charter operators appreciate their autonomy, and while most do enjoy wider latitude over personnel decisions than large urban school districts that serve as their hosts, most do not necessarily base the majority of their hiring, firing and compensation decisions on student test scores and test scores alone. And most don’t seem very interested in being compelled to do so – adopting a one size fits all evaluation model. Apparently, they are also relatively uninterested in disclosing how they evaluate faculty. Here’s a quote from the Albany Times Union.

Carroll, one of the most prominent education reformers in the state, helped write the state’s original charter laws. He said if the charter schools accepted the money, they would lose their current flexibility in the firing and hiring of teachers. He also said charter schools would be forced to disclose their teacher evaluation process, which is now confidential, and that it could become harder to fire an educator deemed ineffective.

http://www.timesunion.com/default/article/No-to-cash-with-a-catch-714008.php

So then, if expanding charters is a major component of reform, and making sure teachers are evaluated by student test scores is a major component of reform, how can this apparent clash be reconciled? It can’t! It seems hypocritical at best to force public school districts to play by a seriously flawed set of teacher evaluation rules and then let charters off the hook? This is especially true if one of the supposed benefits of charter schools is to experiment with creative strategies that may be emulated by traditional public schools and if traditional public schools are expected to improve by competing with charters on a level playing field. I’m with the charter leaders on this one.

UPDATE: Tom Carrol has clarified his comments here: http://www.nyfera.org/?p=2827, where he attempts to explain that charters are opposed to the teacher evaluation requirements not because charters oppose the ideas behind using data to evaluate teachers, but that charters oppose having the state education department mandate how that data should be used in evaluations:

SED simply has no authority to set thresholds for the use of data in teacher evaluations in charter schools.  Nor do they have the authority to require us to group teachers by four categories, or require such annual evaluations to be “a significant factor” for “promotion, retention, tenure determination and supplemental compensation.”  Nor do they have the authority to require charters to pursue “the removal of teachers and principals receiving two consecutive annual ratings of ‘ineffective’ after receiving supports from improvement plans.”

Carrol’s clarification coupled with his unsubstantiated claim that charters are already doing these things really doesn’t change my point above – that it remains hypocritical to hoist these deeply problematic policies on traditional public schools while letting charters off the hook on the basis that charters should be given the flexibility to experiment with alternative strategies and should be exempt from full disclosure regarding those strategies.

Part II: Ideology Clashing with Research

Now, this one is really not an obvious major clash, but rather a more subtle clash between ideology and research embedded in Eric Hanushek’s WSJ editorial the other day.  Embedded in the editorial were comments/claims that I found at least a little disingenuous.

The point of Hanushek’s OpEd was to explain that there is no war on teachers, and that this is really about getting teacher’s unions to negotiate for reasonable changes in contracts that would allow for more expeditious dismissal of the worst teachers – the bottom 5% or so. I’ll admit that I really haven’t seen Hanushek himself outright attacking teachers in his work of late, especially in his actual research on teacher labor markets. Much of it is very good and very useful stuff. That said, it’s hard to deny that many major public figures – talking head tweeters and bloggers for think tanks, etc. – have actually engaged in an all out attack on teachers, the teaching profession and teachers unions.

Setting that broader issue aside, Hanushek’s Op Ed rubbed me the wrong way because of examples he chose to advance his argument, and the extent to which his examples in the Op Ed clash – and clash quite significantly – with his own best recent research. Hanushek summarized his arguments about the non-war on teachers as follows:

What’s really going on is different. President Obama states that we can’t tolerate bad teachers in classrooms, and he has promoted rewarding the most effective teachers so they stay in the classroom. The Los Angeles Times published data identifying both effective and ineffective teachers. And “Waiting for ‘Superman'” (in which I provide commentary) highlighted exceptional teachers and pointed out that teachers unions don’t focus enough on teacher quality.

This is not a war on teachers en masse. It is recognition of what every parent knows: Some teachers are exceptional, but a small number are dreadful. And if that is the case, we should think of ways to change the balance.

http://online.wsj.com/article_email/SB10001424052748703794104575546502615802206-lMyQjAxMTAwMDEwODExNDgyWj.html

So, part of his claim is that it was unjustified for teachers unions – not teachers mind you – but their unions to object so loudly when the LA Times merely – in the public interest – revealed data which validly (implied above, by absence of disclaimers) identified, labeled and named effective and ineffective teachers.

Wait… isn’t it Eric Hanushek’s own research and writing that highlights problems with using value-added measurement to evaluate teachers where non-random student assignment occurs (which is pretty much anywhere)? For me, it was my familiarity with his work that led me to explore the biases in the LAT model that I’ve written about previously on this blog.  In that same post, I explain why I am more inclined to accept Jesse Rothstein’s concerns over the problems of non-random assignment of students than to brush those concerns aside based on the findings of Kane and Staiger.  Hanushek provides a compelling explanation for why he too places more weight on Rothstein’s findings.

Correct me if I’m wrong, but isn’t it possible that teachers and their union were in an uproar at least partly because the LA Times released highly suspect, potentially error ridden and extremely biased estimates of teacher quality? And that the LA Times misrepresented those estimates to the general public as good, real estimates of actual teacher effectiveness?

Yes, much of Eric Hanushek’s recent writing does advocate for some reasonable use of value-added estimates for determining teacher effectiveness, but he usually does so while giving appropriate attention to the various caveats and while emphasizing that value-added estimates should likely not be a single determining factor. He notes:

Potential problems certainly suggest that statistical estimates of quality based on student achievement in reading and mathematics should not constitute the sole component of any evaluation system.

http://edpro.stanford.edu/hanushek/admin/pages/files/uploads/HanushekRivkin%20AEA2010.CALDER.pdf

Yet, that’s just what the LA Times did, and without even mentioning the caveats!

Isn’t it a bit of an unfair assertion, given Hanushek’s own research and writing on value-added estimates, to claim that LA teachers and their union were completely unjustified in their response to the LA Times?

Part III: Ideology Clashing with Reality

There exists at least one segment of the truly reformy crowd that believes deeply in second major ideology laid out at the beginning of this post – that if we can simply close failing urban schools (the whole district if we have to!) and let charters proliferate, children in the urban core will have many more opportunities to attend truly good schools. Yes, these reformers throw in the caveat that we must let only “good” charters, “high performing” charters start-up in place of the failing urban schools. And when viewing the situation retrospectively, these same reformy types will point out that if we look only at the upper half of the charters, they are doing better than average. Yeah… Yeah… whatever.

One long-term research project that has interested me of late is to look in-depth at those “failing” urban school districts that over the past decade have had the largest shares of their student population shift to charter schools – that is, the largest charter market share districts. Here is link to the Charter Market share report from the National Alliance for Public Charter Schools: http://www.publiccharters.org/Market_Share_09

It would seem that if we adopt the reformy ideology above, that if we identify those districts with the largest charter market shares, those districts should now be models for high quality, equitably distributed educational opportunities. We should eventually see sizeable effects in the achievement and attainment of children growing up in these cities, we should see quality of life increasing dramatically, housing values improving with an influx of families with school-aged children – a variety of interesting, empirically testable hypotheses, which I hope to explore in the future.

In the mean-time, however, we have new and interesting descriptive information from a report from the Ewing and Marion Kauffman Foundation in Kansas City, focused on educational opportunities in Kansas City. Kansas City is #4 on charter market share, according to the National Alliance report, and rose to that position much earlier in the charter proliferation era than other cities. As a result, by reformy logic, Kansas City should be a hotbed for educational opportunity for school-aged children – after years of previously throwing money down the drain in the Kansas City Missouri Public School District (many of these claims actually being Urban Legend).

In Kansas City, the reality of charter expansion has clashed substantially with the reformy ideology. Arthur Benson in a recent Kansas City Star Op Ed, noted:

Charters have subtle means for selecting or de-selecting students to fit their school’s model. The Kansas City School District keeps its doors open to non-English speakers and all those kids sent back from the charter schools. In spite of those hurdles, Kansas City district schools across the board out-perform charter schools. That is not saying much. We have until recently failed 80 percent of our kids, but most charters fail more.

I was initially curious about Benson’s (a district board member and attorney) claims that charters have done so poorly in Kansas City. Could it really be that the massive expansion of charter schools in Kansas City has done little to improve and may have aided in the erosion of high quality educational opportunities for Kansas City children?

The recent Kauffman Foundation report draws some similar conclusions, and Kauffman Foundation has generally been an advocate for charter schools. The report classifies district and charter schools into groups by performance, with level 4 being the lowest, and level 1 being the only acceptable group.

  • Level I- A school that met or exceeded the state standard on the MAP Communication Arts and Mathematics exams in 2008-2009.
  • Level II- A school that scored between 75 and 99 percent of the state standard on the MAP Communication Arts and Mathematics exams in 2008-2009.
  • Level III– A school that scored between 50 and 74 percent of the state standard on the MAP Communication Arts and Mathematics exams in 2008-2009.
  • Level IV– A school that scored below 50 percent of the state standard on the MAP Communication

Among other things, the report found that charter operators had avoided opening schools in the neediest neighborhoods. Rather, they set up shop in lower need neighborhoods, potentially exacerbating disparities in opportunities across the city’s zip codes. The report recommended:

A strategy for charter school growth should be developed by Kansas City education leaders. Charter schools should only be approved by DESE if they can demonstrate how they intend to fill a geographic need or a specific void in the communities they intend to serve.

Regarding charter performance more generally, the report noted:

In many communities charter schools are a model that increases students’ access to better public schools, but the majority of charter school students (5,490 or 64.7 percent) are in a Level IV school. Many of Kansas City’s charters have existed for 10 years and are still not able to reach even half of state standard.

Now, I’m not sure I accept their premise that in many communities this actually works – and that it just went awry for some strange reason in Kansas City. That said, the reality in Kansas City, by the authors own acknowledgment is in sharp contrast with the reality the authors believe exists in other cities.

One implication (not tested directly) of this report is that the massive charter school expansion that occurred in Kansas City may have done little or nothing to improve the overall availability or distribution of educational opportunities for children in that city and may have actually made things worse.

Isn’t it strange how we hear so little about these things as we look to replicate these models of great reformy success in other cities of comparable scale such as Newark, NJ?

Money and the Market for High Quality Schools

This post is a revised version of my previous post – If money doesn’t matter…

Here is a draft set of slides to accompany this post: Resource Heterogeneity across Sectors

The theme du jour is that reform (very narrowly defined reform), not money will fix our schools. We’re already spending a lot, the pundits say. Too much in fact, for what we’re getting. We need more charter schools – which obviously do more with less – we need to treat teachers like workers in the private sector (?) by publicly ranking them based on their students’ test scores – and in general, we need to adopt “market” oriented strategies. But…

If money doesn’t matter then why do private independent schools (market driven schools?) spend, on average, so much more per child than nearby public schools?

First off, I am a supporter of private independent schools and former teacher in a private independent school in New York City – An exceptional school where tuition is now about $35,000 per child (where tuition covers only a portion of expense) in a city where the public system is being chastised by politicians and the popular media for spending about $20,000 per child. This despite the fact that the city school system must serve a more diverse and complex student population than the very selective private school where I taught.

About a year ago, I published a study on the private school marketplace in which I compiled the IRS financial filings of about 1,600 private schools around the country. And what did I find in this study? Among many other things, I found that private independent schools, a relatively large diverse sector of schools which includes many elite schools, and also some pretty average ones, spent on average 196% of public school average on same labor market (excluding boarding schools).

I also found that the pupil to teacher ratios in private independent day schools are about 8.8/1 (consistently from 2000 to 2008) compared to those of public schools at about 16.7/1 over the same period.

That is, private independent schools – ON AVERAGE – not just the elite of the elite – spend nearly double what public schools in the same area spend, and private independent schools leverage that money to purchase nearly double the teachers per child, offering much smaller class sizes, deeper and broader elective options, music, arts and other “frills” many public schools have seen evaporate with recent budget cuts.

I am by no means criticizing the choice to provide one’s own child with a more expensive education. That is a rational choice, when more expensive is coupled with substantive, observable differences in what a school offers. I am criticizing the outright hypocritical argument that money wouldn’t/couldn’t possibly help public schools provide opportunities (breadth of high school course offerings, smaller class sizes) more similar to those of elite private independent day schools, when this argument is made by individuals who prefer private schools that spend double what nearby public schools spend.

Private School Spending Study: http://nepc.colorado.edu/publication/private-schooling-US

If money doesn’t matter then why do venture philanthropists continue to throw money at charter schools while throwing stones at traditional public schools?

The standard rhetoric, touted in the media these days is that charter schools are not only doing better than traditional public schools, but that they are doing so with far fewer financial resources. The reality is that charter schools have widely varied resources, from state to state and even from block to block within New York City. Let’s focus on New York City charters for a moment, because New York City charter schools have received so much media attention.

A New York City Independent Budget Office report suggested that charter schools housed in public school facilities have comparable public subsidy to traditional NYC public schools, but charter schools not housed in public school facilities have to make up about $2,500 (per pupil) in difference. In forthcoming report, I explain how the much lower need populations served by NYC charter schools, compared to nearby NYC traditional public schools, more than offsets this difference.  That is, from the start, NYC charter schools are on relatively level financial playing field with the traditional public schools against which they supposedly compete. In fact, charters provided with physical space have a head start, and serve fewer low income children, few or no ELL children and fewer children with disabilities.

And then there’s the philanthropy. Kim Gittleson of Gotham Schools points out that in 2008-09, NYC Charter schools raised an average of $1,654 per pupil through philanthropy. In 2009, Venture Philanthropists granted over $30 million to 77 NYC charter schools, excluding major gifts to management organizations associated with many of the NYC charter schools.

Some NYC charter schools raised more than $8,000 per pupil, and depending on how you calculate it, Harlem Children’s Zone comes in as high as $60,000 per pupil. As a result, some charters – those most favored by venture philanthropists – spend on a per pupil basis much more than traditional NYC public schools.

One might argue that the Venture Philanthropists are trying to spend their way to success – To outspend the public schools in order to beat them!

In fact, a recent study funded by the New Schools Venture Fund indicated “The average CMO relies on philanthropy for approximately 13 percent of its total operating revenues, but many CMO central offices could not exist today without philanthropy.” That is, they need this level of infusion just to stay afloat, running each year in the red, with no sign of break even years in the near future.

But here’s the disconnect – These same Venture Philanthropists – who are committed to spending whatever it takes on charters in order to prove they can succeed, can be frequently heard arguing that public schools a) don’t need and b) could never use effectively any more money. They are trying to argue that charters are doing more with less, when some are doing more with more, others less with less, and some may be doing more with less, and others are actually doing less with more. Shouldn’t traditional public schools be given similar opportunity to do more with more? Blasphemy! Eh? And don’t give me that … “we’ve already tried that and it didn’t work” claim. I’ll gladly provide the evidence to refute that one!

If money doesn’t matter then why do affluent – and/or low poverty – suburban school districts continue in many parts of the country to dramatically outspend their poorer urban neighbors?

Last but not least, why do affluent suburban school districts in many states continue to far outspend poor urban ones? If there is no utility to the additional dollar spent and/or no effect produced by that additional dollar then why spend it?

In a recent article, co-author Kevin Welner and I point out that many pundits have prematurely argued that states have done away with – erased – differences in resources across wealthy and poor districts (article here: http://epaa.asu.edu/ojs/article/view/718) Really, anyone with a grain of information on this topic knows this assumption to be patently false. See also www.schoolfundingfairness.org.

Among other things Kevin Welner and I point out that nationally, there remains a positive relationship between school district spending per pupil and median household income but some progress was made through the early 1990s. It leveled off since. More importantly, that progress varies widely by state, with some states like New Jersey and Massachusetts providing more support in higher poverty settings, but many like Illinois or New York maintaining systems where affluent, predominantly white school districts continue to far outspend poor urban and urban fringe districts.

For example, in the New York Metropolitan area including only New York State districts (2007-08), lower poverty districts (those with fewer than 10% children below the poverty line) had state and local revenues per pupil ON AVERAGE, at about $23,000 to $24,000 per pupil, compared to those with over 20% poverty (census poverty rate) at just over $18,000 per pupil in state and local revenues in that same year. Yet pundits pick the $18,000 per pupil number out of context, call it too high, and argue they should get no more! No more I tell you! The waste is egregious! Kevin Welner and I identify 9 downstate suburban districts that spent more than $10,000 more per pupil than New York City in 2007-08.

If the waste in New York City, or in Newark, New Jersey is so gosh darn egregious – if we’re spending way beyond reasonable levels in poor urban districts, what about those districts spending so much more on kids who would do just fine on so much less? Isn’t that just a massive freakin’ waste? The people in these communities don’t seem to think so.

In conclusion…

Here’s the thing – I don’t believe that private independent schools or affluent local public school districts are just throwing money away. I believe they are trying to provide a high quality product to consumers who demand such a product and who expect such a product, be it through a system of local public financing or through a private market based system.

That’s the interesting twist in all of this. The “reformers” who are choosing expensive private schools for their own children and throwing money at charters are invoking the language of “market based reforms” for traditional public schools – market based reforms as a substitute for more money – because market based reforms will ALWAYS drive down per pupil spending. That’s what competition does, right?

Well, the one set of schools in this mix that are arguably most responsive to “market pressures” are the private independent schools. The schools most responsive to market pressures are the ones that a) spend the most, b) have the smallest class sizes and seem to use small class size in particular as a primary selling point, and c) I would venture to guess are least likely to be moving down the road of evaluating all of their teachers on the basis of test scores alone (most actually have relatively traditional experience driven step-scales). Yeah… yeah… but those are the luxury market products? Are small class sizes and diverse high school curriculum luxuries that should be reserved for only the few? I find this argument most offensive.

Really good education is expensive – and far more expensive than “reformers” are willing to admit or understand. If the “reform movement” is really about mimicking successful business models, these entrepreneurs should be paying close attention to the money being spent producing a high quality product – benchmarking against the “best” public and private schools, and then realizing that achieving comparable outcomes with more needy student populations will cost more – a lot more – not less. For some reason, in this case, they’ve ignored that conversation entirely!


Intellectual Pathologies of the Reformy World (Kevin vs. Kevin)

Yesterday, a colleague and coauthor on two recent articles – Kevin Welner (U. of Colorado) – wrote a scathing critique of the manifesto on fixing urban schools that was released last week by several large city superintendents.

Kevin Welner’s commentary can be found here: http://voices.washingtonpost.com/answer-sheet/guest-bloggers/manifesto-should-be-resignatio.html

The manifesto can be found here: http://www.washingtonpost.com/wp-dyn/content/article/2010/10/07/AR2010100705078.html

Kevin Carey notes in his critique of Kevin Welner:

I highlight this because it’s crucial to understanding the worst intellectual pathologies of the education establishment. People like Welner don’t just think that Joel Klein, Michele Rhee, Andres Alonso, and Arlene Ackerman are making bad decisions in the course of helping poor children learn. Welner believes that by asserting that poor children can learn, the superintendents are hurting the cause of making poor children less poor. While many people believe this, most choose not to say it so clearly.

http://www.quickanded.com/2010/10/the-supposed-trouble-with-helping-poor-students-learn.html

I urge you to take a look at what Kevin Welner actually said in his commentary. The centerpiece of Kevin Welner’s argument was that the superintendents and others behind the manifesto were making a strong sales pitch for fast-tracking education reform strategies for which the research base is mixed at best. Kevin Welner asks:

Are these adults acting responsibly when they advocate for even more test-based accountability and school choice? Over the past two decades, haven’t these two policies dominated the reform landscape – and what do we have to show for it? Wouldn’t true reform move away from what has not been working, rather than further intensifying those ineffective policies? Are they acting responsibly when they promote unproven gimmicks as solutions?

Are they acting responsibly when they do not acknowledge their own role in failing to secure the opportunities and resources needed by students in their own districts, opting instead to place the blame on those struggling in classrooms to help students learn?

And Kevin Welner summarizes the manifesto as follows:

Move money from neighborhood schools to charter schools!
Make children take more tests!
Move money from classrooms to online learning!
Blame teachers and their unions – make them easier to fire!
Tie teacher jobs and salaries to student test scores!

Explaining:

None – literally NONE – of these gimmicks is evidence-based.

I tend to agree that the findings on expansion of charters are mixed at best, and that tying teacher ratings to test scores is deeply problematic. Perhaps what irked Kevin Carey most here, is that he has convinced himself, through exceedingly flimsy logic, that he Kevin Carey is right, and that other Kevin, Kevin Welner is simply wrong on these points. Allow me to bring you back to a series of recent comments by Kevin Carey that display his completely distorted understanding of research on charters (and implications for policy) and the usefulness of value-added modeling to rate teachers.

Kevin Carey on Charters

Here’s a recent quote from Kevin Carey, attacking the civil rights framework on whether the evidence supports expansion of charter schools.

Here’s the problem: the contention that charters have “little or no evidentiary support” rests on studies finding that the average performance of all charters is generally indistinguishable from the average regular public school. At the same time, reasonable people acknowledge that the best charter schools–let’s call them “high-quality” charter schools–are really good, and there’s plenty of research to support this.

http://www.quickanded.com/2010/08/evidence-and-the-civil-rights-group-framework.html

I have noted previously, here, that I find this to be one of the most patently stupid arguments I think I’ve seen in a long time.

To put it in really simple terms:

THE UPPER HALF OF ALL SCHOOLS OUTPERFORM THE AVERAGE OF ALL SCHOOLS!!!!!

or … Good schools outperform average ones. Really?

Why should that be any different for charter schools (accepting a similar distribution) that have a similar average performance to all schools?

This is absurd logic for promoting charter schools as some sort of unified reform strategy – Saying… we want to replicate the best charter schools (not that other half of them that don’t do so well).

Yes, one can point to specific analyses of specific charter models adopted in specific locations and identify them as particularly successful. And, we might learn something from these models which might be used in new charter schools or might even be used in traditional public schools.

But the idea that “successful charters” (the upper half) are evidence that charters are “successful” is just plain silly.

Kevin Carey on Value-Added Teacher Ratings

In the New York Times Room for Debate series on value-added measurement of teachers, Carey argued that Value-added measures would protect teachers from favoritism. Principals would no-longer be able to go after certain teachers based on their own personal biases. Teachers would be able to back up their “real” performance with hard data. Here’s a quote:

“Value-added analysis can protect teachers from favoritism by using hard numbers and allow those with unorthodox methods to prove their worth.” (Kevin Carey, here)

The reality is that value-added measures simply create new opportunities to manipulate teacher evaluations through favoritism. In fact, it might even be easier to get a teacher fired by making sure the teacher has a weak value-added scorecard. Because value-added estimates are sensitive to non-random assignment of students, principals can easily manipulate the distributions of disruptive students, students with special needs, students with weak prior growth and other factors, which, if not fully accounted for by the VA model will bias teacher ratings. More here!

Kevin Carey also claims as a matter of accepted fact, that VA measures “level the playing field for teachers who are assigned students of different ability.” This statement, as a general conclusion, is wrong.

  1. VA measures do account for the initial performance level of individual students, or they would not be VA measures. Even this becomes problematic when measures are annual rather than fall/spring, so that summer learning loss is included in the year to year gain. An even more thorough approach for reducing model bias is to have multiple years of lagged scores on each child in order to estimate the extent to which a teacher can change a child’s trajectory (growth curve). That makes it more difficult to evaluate 3rd or 4th grade teachers, where many lagged scores aren’t yet available. The LAT model may have had multiple years of data on each teacher, but didn’t have multiple lagged scores on each child. All that the LAT approach does is to generate a more stable measure for a teacher, even if it is merely a stable measure of the bias of which students that teacher typically has assigned to him/her.
  2. VA measures might crudely account for socio-economic status, disability status or language proficiency status, which may also  affect learning gains. But, typical VA models, like the LA Times model by Buddin tend to use relatively crude, dichotomous proxies/indicators for these things. They don’t effectively capture the range of differences among kids. They don’t capture numerous potentially important, unmeasured differences.  Nor do they typically capture classroom composition – peer group – effect which has been shown to be significant in many studies, whether measured by racial/ethnic/socioeconomic composition of the peer group or by average performance of the peer group.
  3. For students who have more than one teacher across subjects (and/or teaching aides/assistants), each teacher’s VA measures may be influenced by the other teachers serving the same students.

I could go on, but recommend revisiting my previous posts on the topic where I have already addressed most of these concerns.

Intellectual pathologies?  Pot… kettle?


On School Funding Fairness

I’ve been toying around for a while on this blog with different ways to compare state school finance systems. This new website presents a summary of much of that playing:

http://www.schoolfundingfairness.org/index.htm

After much discussion and debate, we landed on the following four indicators.

The Fairness Measures
All 50 states are evaluated on the basis of four separate, but interrelated, fairness measures:

  • Funding Level: Using figures adjusted to account for a variety of interstate differences, this measure allows for a comparison of the average state and local revenue per pupil across states. States are ranked from the highest to lowest per pupil funding.
  • Funding Distribution: This measure shows whether a state provides more or less funding to schools based on their poverty concentration. States are evaluated as “regressive”, “progressive”, or “flat” and are given letter grades that correspond to their relative position compared to other states.
  • Effort: This measures differences in state spending relative to the state’s fiscal capacity. States are ranked according to the ratio of state spending on education to per-capita gross domestic product.
  • Coverage: This measures the proportion of school-age children attending the state’s public schools and also addresses the income disparity between families using private, rather than public, schools. States are ranked according to both the proportion of children in public schools and the income ratio of private and public school families.

It is important to understand that two of these indicators are much more in control of the states than others – Effort and Funding distribution, or Fairness. States control the amount financial effort they put into their schools, as a percent of their capacity. States have less control over their overall funding level produced by that effort. I encourage you to look carefully at differences between states like Louisiana and Tennessee, compared to Mississippi. The first two simply don’t put up the effort. Hence my constant lambasting of Tennessee on this blog, especially in the context of RttT.

Coverage and funding level are not as controlled by states, but that’s not to say they are not significantly controlled by states. Funding levels vary about 50/50 on the basis of state wealth and on the basis of state effort. Effort seems to matter as much as wealth in predicting state spending levels, and that makes sense. Hence we grade on effort.

Coverage is included for a few reasons, and is included along with the ratio of family income of those in and not in the public education system. First, coverage is included because for many states this indicator shows just how many kids we are leaving out of our equity analysis by comparing revenues across only the public system. In a handful of states, the excluded share is around 20% and in some of those states that 20% are from much higher income households – likely increasing the “regressiveness” of the system as a whole (public/private schooling). Second, we include coverage because long term, systemic deprivation of the public system can, in fact, lead to significant flight from the public system. That should not be ignored and should not be treated, as one reader of a previous post argued (comments section), as a smart state policy decision toward further reducing long run costs- by encouraging more affluent families to independently finance their child’s education. Call it a value-laden decision, but we do not accept the argument that depriving the public education system to the point where more kids opt out, so we don’t have to use tax dollars to pay for them, is smart policy.

I also encourage readers not to try to make too much of the between state comparisons of overall spending level. I discuss many angles on these comparisons in a recent post: https://schoolfinance101.wordpress.com/2010/10/04/state-ranking-madness-who-spends-mostleast/ The bottom line is that it’s really hard to make reasonable comparisons of the cost differences of operating schools in Vermont versus Nevada. However, the within region comparisons may be more useful.

I especially encourage comparisons among the “profiles” or those sloped lines among states sharing regions. The New York/New Jersey profile comparisons are particularly interesting. New York affluent suburban districts have far more resources than New Jersey affluent suburban districts, but for poor urban districts, the differences flip.

Be sure also to check out the updated tables with the 2007-08 NCES data.

The research question that wasn’t asked

Recent discussions of the Vanderbilt University study on the effect of merit pay in Nashville raised a common and important issue pertaining to education policy research – or any research for that matter – What about the question that wasn’t asked? Or how important really is the question that was asked?

In the case of the Vanderbilt merit pay study, the researchers essentially asked whether providing sizable financial bonuses to randomly selected teachers could motivate those teachers to try harder and ultimately produce better student outcomes than teachers randomly selected to be in the group that could not get bonuses. That is, does the merit pay serve to make one randomly selected group of teachers produce better student outcomes than a control group?

Pundits quickly leaped on the question NOT ASKED – which was whether or not changing teacher compensation structures more generally – making teaching a profession based on rewards for performance or a profession where one could increase income over time by being a high performer would ultimately change the quality of individuals who would enter the teaching workforce.

That is, the study asked whether financial incentives could change the behavior of those already in the system, but not whether the existence of performance incentives would change those who choose to be in the system.

Now, when “reformy” types pointed to this question NOT ASKED, they also seemed to uniformly imply that we know the answer to the question not asked – and that is – “of course this would encourage better teachers to enter the labor market.” You know what – the question wasn’t asked. It wasn’t tested and we certainly do not know this to be the answer. For now, the answer is “we don’t know,” and it is likely fair to say that the answer is “it depends, on a variety of factors including how compensation is altered, the risk/reward ratios, etc. etc. etc.”

This brings me to a comment made by Andrew Rotherham in his recent Time Magazine post:

For example, it’s clear from abundant research that paying teachers only on the basis of their degrees and years of experience is not in the best interest of students or teachers. As the National Council on Teacher Quality, a research and policy organization whose board of directors I chaired for several years, put it, “the evidence is conclusive that master’s degrees do not make teachers more effective.”

Now, I think even this statement is a bit, well, overstated. The “research” Rotherham seems to draw on here (and NCTQ dreadfully overstates) is research that asked the following questions:

  1. Do teachers who hold general masters degrees, versus those who do not, scattered across a variety of settings, show differences in the average outcome gains of their students?
  2. Do teachers at varied levels of experience, scattered across a variety of settings, show differences in the average outcome gains of their students?

The first of these questions was beaten into the ground over and over in the 1990s, often using data from the National Education Longitudinal Studies (NELS ’88) with many of the studies showing no relationship between holding a masters or not and student outcomes, and at least a few showing positive effects of holding a content area masters in math/science (I’m doing this largely from memory).

The second of these questions has been addressed in a number of recent analyses, as well as some older ones. More recent studies have generally evaluated the average student value added ratings of teachers by their experience levels. Many of these studies find that teachers in their first two to three years tended to show smaller student achievement gains than teachers in their 4th, 5th or 6th years, but after that, things really kind of level off. Here’s an example of such analysis: http://www.urban.org/UploadedPDF/1001455-impact-teacher-experience.pdf

Interestingly, pundits pushing so hard for major changes to the risk/reward structure of teacher compensation who are so quick to point out the question not asked in the Vanderbilt merit pay study fail to recognize that similar labor market questions were never asked in these studies either.  Researchers asked whether teachers with certain attributes had better student outcomes than teachers with different attributes. As far as I recall, no one ever asked whether differential compensation on the basis of these attributes produced any desirable or undesirable labor market effects – changes to the applicant pool, etc.

Studies of the association between different levels of experience and the association between having a masters degree or not and student achievement gains have never attempted to ask about the potential labor market consequences of stopping providing additional compensation for teachers choosing to further their education – even if only for personal interest – or stopping providing any guarantee that a teacher’s compensation will grow at a predictable rate over time throughout the teacher’s career.

Many, like Rotherham but even more so, NCTQ, present this as a “research given.”  That clearly, it’s just dumb to pay teachers more who possess attributes we know are not associated with student achievement differences (across teachers). Is it possible, however, that changing these conditions could have significant labor market consequences? Perhaps good… but equally likely… unintended negative consequences.

Yes, teachers with any old masters degree or teachers with more than 10 years behind them might not, on average, be “measurably more productive.” But does the option to pay and recruit more experienced teachers or teachers with masters’ degrees enhance the likelihood that a district can attract teachers who are actually better teachers? I’m not so sure that the answer to this question unasked is so obvious that we need not ask it. So let’s stop pretending that it is.