In a recent Room for Debate in the New York Times, I argued that focusing on test scores to evaluate teacher quality allows politicians and the public to ignore the overwhelming source of educational problems—poverty. While this stance is well supported by evidence and many leading scholars in education, sociology, and economics, the questions my response elicited online as well as the questions that were not raised are an ideal opportunity to examine the many misconceptions around education reform and teacher quality.
First, many people have trouble accepting an irrefutable fact: Single data points (such as test scores) are overwhelmingly most highly correlated with any student's out-of-school circumstances. For one easily accessible example, I encourage everyone to examine the wealth of data on the SAT provided by the College Board. Every year the SAT has been administered, scores have been most strongly and directly correlated with "Family Income" and "Highest Level of Parental Education."
This pattern of out-of-school factors being the greatest influence on test scores has been found in virtually all testing situations, but with the SAT, this evidence is extremely important because the pool of students taking the SAT are uniquely elite compared to all students, meaning this pool of students is enrolled in the most challenging courses and learning from the most qualified and experienced teachers (see the work of Peske and Haycock, which reveals that children of color, children in poverty, and ELL students are disproportionately taught by teachers who are un- or under-qualified and least experienced).
Test scores, then, are poor evidence for teacher quality because those scores mask what impact teacher quality does have on the lives and learning of children. But this remains ignored by many political leaders, education reformers, and the general public, leading to three very common retorts to any recognition that poverty overwhelms the impact of schools and teachers: (1) Poverty should not be used as an excuse. (2) If some exceptional schools (often charters) can produce high test scores, then why can't they all? And (3) Just test students at the beginning and end of each year to factor out the home conditions of the children.
Those of us identifying that poverty overwhelms the impact of schools and teachers that is captured in test scores are not suggesting that this is evidence of poverty being fatalistic. In fact, our position is a call to address poverty in both society (see Berliner's excellent identification of six out-of-school factors we must address for successful schools) and in schools (again, consider Perske and Haycock's evidence that school perpetuates the inequities found in society through teacher assignment)—instead of repeating the same failed test-and-accountability paradigm we have tried for over thirty years.
Next, many people have fallen into the trap of believing the repeated claims of education "miracles"—found in the Texas miracle, the Chicago miracle, and the Harlem miracle; the popular embracing of the recent documentary Waiting for "Superman" has helped reinforce a common flaw in popular belief: Making the exceptional normal. But more damning than asking everyone to match the characteristics of outliers is that every media-created education miracle I have ever witnessed has been found later to be no miracle at all (see the recent unmasking by Diane Ravitch).
The third question—why not just pre- and post-test each year—is incredibly compelling because examples such as the SAT are addressing single-test situations that seem more likely impacted by external factors. But the evidence on value-added methods (what lay people are usually calling for) shows that a teacher identified as excellent one year may be identified as poor the next; in short, the method is highly volatile.
Part of the problem with any quantified system of evaluating teachers based on student outcomes is that students spend most of their lives outside of school—and any score a child ever produces on a test is the culmination of that child's whole life. To isolate and identify direct causational relationships between any teacher and any student would be far too costly in time and money to justify the effort.
While the challenges above elicited by acknowledging the powerful influence of children's lives on their educational outcomes are persistent, they may not be as important as what we are not asking. While identifying and rewarding teacher quality based in part or entirely on measurable student outcomes is ultimately flawed, we rarely consider that any such system is based on competition—identifying a limited pool of resources for teacher quality and requiring teachers to compete for that pool.
And this may be what is most wrong about pursing test scores to identify teacher quality because education and students always lose when learning is reduced to competition for resources. Teachers and students will thrive in a cooperative and collaborative community of learners; basing teacher quality on student outcomes will pit teacher against teacher, teacher against student with everyone losing.
Education reform is needed, and addressing teacher quality is clearly an important element of that reform. But neither will achieve what we are seeking unless we commit to addressing poverty in the lives of children while reforming our schools to insure that once in the classroom, we stop perpetuating the exact inequities we claim education can overcome.