The research question that wasn’t asked

NCTQ, Reformy, Uncategorized 4 Minutes

Recent discussions of the Vanderbilt University study on the effect of merit pay in Nashville raised a common and important issue pertaining to education policy research – or any research for that matter – What about the question that wasn’t asked? Or how important really is the question that was asked?

In the case of the Vanderbilt merit pay study, the researchers essentially asked whether providing sizable financial bonuses to randomly selected teachers could motivate those teachers to try harder and ultimately produce better student outcomes than teachers randomly selected to be in the group that could not get bonuses. That is, does the merit pay serve to make one randomly selected group of teachers produce better student outcomes than a control group?

Pundits quickly leaped on the question NOT ASKED – which was whether or not changing teacher compensation structures more generally – making teaching a profession based on rewards for performance or a profession where one could increase income over time by being a high performer would ultimately change the quality of individuals who would enter the teaching workforce.

That is, the study asked whether financial incentives could change the behavior of those already in the system, but not whether the existence of performance incentives would change those who choose to be in the system.

Now, when “reformy” types pointed to this question NOT ASKED, they also seemed to uniformly imply that we know the answer to the question not asked – and that is – “of course this would encourage better teachers to enter the labor market.” You know what – the question wasn’t asked. It wasn’t tested and we certainly do not know this to be the answer. For now, the answer is “we don’t know,” and it is likely fair to say that the answer is “it depends, on a variety of factors including how compensation is altered, the risk/reward ratios, etc. etc. etc.”

This brings me to a comment made by Andrew Rotherham in his recent Time Magazine post:

For example, it’s clear from abundant research that paying teachers only on the basis of their degrees and years of experience is not in the best interest of students or teachers. As the National Council on Teacher Quality, a research and policy organization whose board of directors I chaired for several years, put it, “the evidence is conclusive that master’s degrees do not make teachers more effective.”

Now, I think even this statement is a bit, well, overstated. The “research” Rotherham seems to draw on here (and NCTQ dreadfully overstates) is research that asked the following questions:

  1. Do teachers who hold general masters degrees, versus those who do not, scattered across a variety of settings, show differences in the average outcome gains of their students?
  2. Do teachers at varied levels of experience, scattered across a variety of settings, show differences in the average outcome gains of their students?

The first of these questions was beaten into the ground over and over in the 1990s, often using data from the National Education Longitudinal Studies (NELS ’88) with many of the studies showing no relationship between holding a masters or not and student outcomes, and at least a few showing positive effects of holding a content area masters in math/science (I’m doing this largely from memory).

The second of these questions has been addressed in a number of recent analyses, as well as some older ones. More recent studies have generally evaluated the average student value added ratings of teachers by their experience levels. Many of these studies find that teachers in their first two to three years tended to show smaller student achievement gains than teachers in their 4th, 5th or 6th years, but after that, things really kind of level off. Here’s an example of such analysis: http://www.urban.org/UploadedPDF/1001455-impact-teacher-experience.pdf

Interestingly, pundits pushing so hard for major changes to the risk/reward structure of teacher compensation who are so quick to point out the question not asked in the Vanderbilt merit pay study fail to recognize that similar labor market questions were never asked in these studies either.  Researchers asked whether teachers with certain attributes had better student outcomes than teachers with different attributes. As far as I recall, no one ever asked whether differential compensation on the basis of these attributes produced any desirable or undesirable labor market effects – changes to the applicant pool, etc.

Studies of the association between different levels of experience and the association between having a masters degree or not and student achievement gains have never attempted to ask about the potential labor market consequences of stopping providing additional compensation for teachers choosing to further their education – even if only for personal interest – or stopping providing any guarantee that a teacher’s compensation will grow at a predictable rate over time throughout the teacher’s career.

Many, like Rotherham but even more so, NCTQ, present this as a “research given.”  That clearly, it’s just dumb to pay teachers more who possess attributes we know are not associated with student achievement differences (across teachers). Is it possible, however, that changing these conditions could have significant labor market consequences? Perhaps good… but equally likely… unintended negative consequences.

Yes, teachers with any old masters degree or teachers with more than 10 years behind them might not, on average, be “measurably more productive.” But does the option to pay and recruit more experienced teachers or teachers with masters’ degrees enhance the likelihood that a district can attract teachers who are actually better teachers? I’m not so sure that the answer to this question unasked is so obvious that we need not ask it. So let’s stop pretending that it is.

Published by schoolfinance101

Bruce Baker is an Professor in the Graduate School of Education at Rutgers, The State University of New Jersey. From 1997 to 2008 he was a professor at the University of Kansas in Lawrence, KS. He is lead author with Preston Green (Penn State University) and Craig Richards (Teachers College, Columbia University) of Financing Education Systems, a graduate level textbook on school finance policy published by Merrill/Prentice-Hall. Professor Baker has written a multitude of peer reviewed research articles on state school finance policy, teacher labor markets, school leadership labor markets and higher education finance and policy. His recent work has focused on measuring cost variations associated with schooling contexts and student population characteristics, including ways to better design state school finance policies and local district allocation formulas (including Weighted Student Funding) for better meeting the needs of students. Baker, along with Preston Green of Penn State University are co-authors of the chapter on Conceptions of Equity in the recently released Handbook of Research Education Finance and Policy, and co-authors of the chapter on the Politics of Education Finance in the Handbook of Education Politics and Policy and co-authors of the chapter on School Finance in the Handbook of Education Policy of the American Educational Research Association. Professor Baker has also consulted for state legislatures, boards of education and other organizations on education policy and school finance issues and has testified in state school finance litigation in Kansas, Missouri and Arizona. He is a member of the Think Tank Review Panel, a group of academic researchers who conduct technical reviews of publicly released think tank reports on education policy issues.

Published