Thinking (& Writing) About Education Research & Policy Implications

Education reporters out there… here are a few thoughts for you as you embark on whatever may be your next article pertaining to an education research study.

FIRST, do a Google Scholar (easiest lit search around!) search on the topic in question to see what other peer reviewed an non-peer reviewed stuff has been written on the same topic? And more specifically, if you are reporting on a “work in progress,” or non-peer reviewed recent release, compare the a) methods used and b) phrasing of major conclusions, to those used in the peer reviewed stuff. While peer review isn’t a be all and end all for research quality, methods do tend to get refined in the process, and junky methods often (though not always) get filtered out or substantively improved (it’s all relative)! More complicated methods aren’t always better. Good authors can explain more complicated methods in reasonable terms.

These next two are perhaps even more important… and require somewhat less technical background…

SECOND, stop, take a breath and revisit your basic knowledge of how schools work – how they are set up – etc. How classrooms are organized – how kids and teachers are sorted across classrooms, schools, neighborhoods, etc.  Ponder how classrooms are organized and how those classrooms may differ from one school to the next, one town or city to the next. Scribble out pictures of “how schools work,” how a child’s day, week, year – inside and outside of school is organized…AND THEN, ONLY THEN, start pondering the possible implications of the study.

THIRD, while pondering the implications of the study, make yourself a list of major current policy agendas and ask yourself – what the heck might any of this mean, when it comes to, say, studies of the effectiveness of charter schools? The effect of charter expansion? Or the usefulness of test-score based measures for evaluating teacher effectiveness?

One recent example that comes to mind is the reporting on a report (well, actually a series of them) from the Hamilton Institute.  Specifically, The Boston Globe covered the portion of the report where one of the report author’s Michael Greenstone indicated that:

High-income families have always invested more in education, but they now spend seven times more a year on average than a low-income family, up from four times in the 1970s, according to the report, coauthored by MIT economics professor Michael Greenstone. These families now spend as much as $9,000 annually on private tutoring, SAT prep courses, computers, and other activities, compared with about $1,300 for low-income families. (cited from the Boston Globe)

The (rather unfulfilling) policy implications punchline(s) from the Boston Globe article were:

For example, said Greenstone, simplifying financial aid applications and providing low-income families help in filling them out could increase college enrollment by about 8 percentage points at a cost of less than $100 a student.

Another recent study found that mailing high-achieving, low-income students personalized information on their college options nudged students to apply to better schools.

Surely, a seven fold difference in private contributions to children’s learning between richer and poorer families has broader implications than this? Right?

Actually, this kind of disparity, and knowing how richer and poorer kids and their schools are organized, has potential ripple effect implications across nearly everything we study in education policy research.   Think about this – just a little bit – from a very basic and practical standpoint.

Wealthier families are adding up to $9k annually to the educational expenditures on their children, compared to $1.3k for less wealthy families.  So, even if these two groups lived in similar towns and attended “equally” funded schools, we’d have a substantial disparity in the financial inputs to their education. Now, if all of this additional spending is pointless, and, for example, doesn’t in any way contribute to improved test scores, then perhaps it’s a non-issue when we consider other implications for popular policy research. But, to the extent that this personal expenditure matters at all, then it has important ripple effects across numerous types of studies, pertaining to current favored policy topics.

For example, if teachers are going to be evaluated on the basis of student test score gains, and those tests are to be given annually, wouldn’t it be better to be the teacher of kids whose parents are spending more (assuming they are choosing wisely) on after school, weekend… and especially SUMMER academic opportunities? Seriously – first consider (jot it down/back of the napkin) how many hours per day for a 185 day school year a kid has contact with her algebra teacher.  Then add up the hours for a typical KUMON program after school or on weekends. Add in all of those summer days, and potential access to a plethora of interesting summer academic & enrichment programs. 45 minutes a day for 185 days is a relatively small portion of a child’s life over the course of a year. Doesn’t take any heavy statistical lifting to figure that out. Just stepping back and think about how kids’ lives and schools are organized.

And is that 45 minutes a day in a class of 35 (dividing the teacher’s attention by 35) really equivalent to 45 minutes in a class of 16? And which kid is more likely in which class? (depends somewhat on state context).

There’s already a substantial body of literature validating substantial summer achievement growth differences by income status. Quite honestly, if our best value-added measures and growth percentile measures aren’t picking up such large, non-random, non-school investments in student learning – if these investments don’t affect the model results – it may just be because the models and measures on which they are based are crap.

It turns out that this differential investment by parents in out of school opportunities not only compromises how we think about per pupil spending differences across children, but it also may blow a pretty big hole in how we interpret a whole lot of other policy research & policy recommendations.

A second example, which I have discussed previously, is reporting on the much discussed CREDO studies of charter school “effects” on student achievement gains.   These studies really require that we ponder how school systems work and how kids sort (as well as how we measure who is similar to one another). Otherwise, we miss some really, really, important points.

First, I’ve explained previously through pictures that studies characterized as “randomized” lottery studies of charter schools really aren’t randomized, which can easily be seen by sketching out where, in the process randomization occurs (lottery).  A true randomized study would take a representative population, and randomly put half in a charter school, and half in a control (whatever that may be) school. Like this:

But a lottery study starts with a sample of those who entered the lottery, which may or may not be representative of the total population – but in theory they were/are all similarly motivated to enter a lottery.  But it’s only the lottery that’s randomized. Not the peer group into which the kids fall when they finally end up at their assigned school. Like this:

So who cares, if they are supposed otherwise similar kids (of course, as I’ve noted, the measures are often insufficient for defining them as such)?  Well, let’s ponder again how schools work and how we evaluate the “effects” of a school on a kid. What’s in a school, after all?

Bricks & mortar, materials, supplies and equipment, yes.

Teachers, yes.

Other school staff, check.

And other kids! Check!

The “effect” of a school as measured in most studies of this type are the “effect” on measured test score changes during a given time period, of all of this stuff – and for that matter – any and all outside of school stuff that goes on during this same time period. And that includes the peer group. And a substantial body of research supports that peer groups matter for student outcomes.

The average current achievement level of the peers affects individual student’s outcomes.[1]

In other words, cream-skimming and/or selective attribution, to the extent it exists and to the extent it affects peer groups, matters (on both the up, and down side)[2] – in this type of study, which considers any and all school conflated factors to contribute to measured school effects.

This is not a condemnation of the CREDO method, but rather a limitation (I might condemn the extent that they ignore and obfuscate this point). It’s really hard to sort out the peer, from teacher or school effect. They’re all conflated. And guess what… all of this stuff then relates back to those huge differences in which kids’ families spend more on their outside of school education! It would certainly be a huge stretch to suggest that positive effects found for a charter school, or charter schooling, using this method tell us anything about the relative effectiveness of charter versus “other” school teachers.

Then there’s the issue  of how these CREDO type studies frequently address (read: brush off) the issue of cream-skimming. First, many use measures insufficient to actually capture cream-skimming (calling all special ed kids, or all “low income” kids equal, when they’re not, and when they may not be randomly sorted as either individuals or peers).

Second, they often set up a deceptive comparison… say… for example… showing that kids who entered charter middle schools from district elementary schools are representative of the total population of their cohort from the district elementary schools. The casual reader then assumes that this means that if the charter applicant and matriculated kids were representative of the populations of the sending schools, then so too must be the kids in the “control” group – district middle schools.

But wait a second, those aren’t the only two pipelines, or options out of feeder schools. Rather, a more complete picture might look like this…

Among kids in those feeder, urban (perhaps) neighborhood elementary schools, when middle school comes along, some may go to district magnet schools that have selective admissions (and thus selective peer groups), some may go to private schools and some may in fact move out to the suburbs. And then there are those who go to the district, “regular” schools – the likely “control” group in CREDO like studies. Do we really think that the kids who sort through each of these various pipelines are similar to one another? Or might comparing against a “feeder” group that sorts in many directions be a little deceptive, at least if it’s done without any acknowledgement of the various directions into which kids sort, and the uneven distribution that may (likely) result from that sorting?

To tie this altogether, it’s also certainly likely that the family contributions to outside of schooling education across these pathways also varies.

So… draw some pictures. Ponder how the system works. Think broadly. Step back & revisit to see if anything might be missing.  Step outside the immediate implications provided by study authors and ask the bigger questions. And with each new study that comes along, don’t forget entirely all those that came before it!

---

[1] Hanushek, E. A., Kain, J. F., Markman, J. M., & Rivkin, S. G. (2003). Does peer ability affect student achievement?. Journal of Applied Econometrics, 18(5), 527-544.

The results indicate that peer achievement has a positive effect on achievement growth. Moreover, students throughout the school test score distribution appear to benefit from higher achieving schoolmates.

Hoxby, C. M., & Weingarth, G. (2005). Taking race out of the equation: School reassignment and the structure of peer effects. Working paper.

We find support for the Boutique and Focus models of peer effects, as well as for a generic monotonicity property by which a higher achieving peer is better for a student’s own achievement all else equal.

Burke, M. A., & Sass, T. R. (2013). Classroom peer effects and student achievement. Journal of Labor Economics, 31(1), 51-82.

…we find that peer effects depend on an individual student’s own ability and on the ability level of the peers under consideration, results that suggest Pareto‐improving redistributions of students across classrooms and/or schools. Estimated peer effects tend to be smaller when teacher fixed effects are included than when they are omitted, a result that suggests co‐movement of peer and teacher quality effects within a student over time. We also find that peer effects tend to be stronger at the classroom level than at the grade level.

[2] Dills, A. K. (2005). Does cream-skimming curdle the milk? A study of peer effects. Economics of Education Review, 24(1), 19-28.

The determinants of education quality remain a puzzle in much of the literature. In particular, no one has been able to isolate the effect of the quality of a student’s peers on achievement. I identify this by considering the introduction of a magnet school into a school district. The magnet school selects high quality students from throughout the school district, generating plausibly exogenous variation in the quality of classmates remaining to those students in the regular schools. I find that the loss of high ability peers lowers the performance of low-scoring students remaining in regular schools.