A trip to the Reformy Education Research Association?

Posted on April 8, 2011



So, as I head off to AERA in New Orleans, I’ve been pondering what it would be like if there was a special education research conference for reformy types.  What would we find at the Reformy Education Research Association, RERA? How would the research be conducted or presented? What kinds of research thinking might we see?

Well, here are a few examples.

Reformy Study #1

First, here’s a table from the widely distributed paper from a team of renowned authors at the Forum on Understanding Core Knowledge in EDucation.

As you can see, the study endeavors to identify the determinants of school failure, in part, to identify those specific policies that must be changed in order to eliminate failing schools from our society. Failing schools are, after all, an abomination.  The researchers ranked New Jersey schools from highest to lowest proficiency rates and took the top and bottom 10%. They then mined the content of the negotiated contractual agreements for each district, looking for key elements of those contracts for explanations for why some districts fail but others perform quite well (as good as Finland!). They also gathered basic demographic data on students, having been dinged by reviewer #3 (an outsider) on their proposal in which they had not included such data. The authors note, however, that including this data did not alter their original conclusions or policy implications.

Conclusion: The cause of some schools failing and others succeeding is clearly the absence of regular use of clear metrics for teacher evaluation and the absence of mutual consent school assignment policies. It is also likely that basing salaries on experience or degree level adds to the dysfunction of low performing schools.

Policy recommendation: Immediately implement a new teacher evaluation system based 50% on student assessment data. Prohibit the use of experience or degree level as a basis for compensation.

Reformy Study #2

In this next study, authors from the Belltower Institute for Technology Education and Modern Enterprise explore the scalability of a nationally recognized model for charter schooling. Specifically, the goal of the study is to determine whether the model, which has received accolades in major newspapers and on network television (Reformy Nation) over the past year, might be a useful model for replacing entire urban school systems.  Table 2 below shows the characteristics of one successful charter school (sufficient data unavailable on the 3 less successful charters in the same network) operating the model, and the characteristics of the urban host district of that charter school. Deliberations are under way in that district to grant the charter operators full control of all schools in the district. Data in the table focus specifically on children in Grades 6 to 8, the only grades served by the charter.

Clearly, the charter not only outperforms the host district schools in grade 6, but by an even larger margin in grade 8, which can only be interpreted (emphasis in original manuscript) as the charter school adding more value to students with each year that they stay (setting aside the possibility that large shares of those students who are nolonger in attendance by 8th grade may have been lower performers).

Again, original analyses included only student assessment scores, and no further information student population characteristics. Amazingly, the original proposal got dinged by the same reviewer #3 as the study above, but reviewers #1 and #2 found the proposal to represent the highest standards of reformy rigor.

The authors continue to maintain that this information is unimportant because the charter populations are necessarily representative of the host district because a lottery is used for admission to the charter. Nonetheless, the authors contend that the reported differences in student populations and cohort attrition are “trivial.”

Conclusion: Clearly, the charter school has proven that it is able to produce far better results than host district schools while serving the very same children (emphasis in original manuscript) as those served by host district schools, and by using its “no excuses” approach.  Further, children’s performance improves the longer they attend the charter school.

Policy recommendation:  Set in place a strategy to turn over all host district schools, across all grade levels to the charter operator.

Reformy Study #3

In the third and final paper, economists from the the Measuring Yearly Advancements in Social Science project released preliminary findings from a massive privately funded study on teacher effectiveness. Specifically, the study endeavors to determine the correlates of effective teaching, in order to guide public school district personnel policies – specifically hiring, retention and compensation decisions. The study involved 22,543 teachers (326 of whom had complete data on all observations) across 6 cities (4 of which failed to provide sufficient data in time for this preliminary release).  Using two years of data on students assigned to each teacher (using only the 4th grade math assessment data, because correlations on language arts assessments were too unreliable to report), the study investigated which factors are most highly related to a TRUE measure of teaching effectiveness – where true “effectiveness” was defined as the contribution of Teacher X, to achievement growth in 4th grade math on the STATE assessment for students S1 – Sy, linked to that teacher in the given year (Equation expressed in Appendix A, pages 69-74).  The same students were also given a second math assessment. School principals conducted observations 5 times during the year and filled out an extensive evaluation matrix based on teacher practices and student – teacher interactions. Students were also administered surveys, as were parents of those students, requesting extensive feedback regarding their perceptions of teacher quality. The correlations are shown in Table 3.

Conclusions & Implications: The strongest correlate of true teaching effectiveness was the estimate of teacher contribution to student achievement on the same test a year later. However, this correlation was only modest (.30). All other measures including effectiveness measures based on alternative tests and student, parent and administrator perceptions of teacher effectiveness were less correlated with the original value-added estimate, thus raising questions about the usefulness of any of these other measures. Because the value-added measure turns out to be the best predictor of itself in a subsequent year, this estimate alone trumps all others in terms of usefulness for making decisions regarding teacher retention (especially in times of staffing reduction) and should also be considered a primary factor in compensation decisions. Note that while it may appear that school administrators, students and their parents have highly consistent views regarding which teachers are more and less effective (note the higher correlations across administrator ratings of teachers, and student and parent ratings), we consider these findings unimportant because none of these perception-based ratings were as correlated with the original value-added estimate as the value-added estimate was with itself (which of course, is the TRUE measure of effectiveness).

About these ads
Posted in: Uncategorized