The most explosive recent education education story is the recent LA Times piece publishing value-added scores and labeling teachers bad as a result. Some applauded what the paper did: isn't value-added a legitimate way of measuring teacher effects? Don't we want to know the good teachers to reward them,and the ineffective teachers we should dismiss? Don't we need student test data for this?
How accurate should data have to be before you're willing to rely on it for high stakes decisions, s bonuses or higher pay for the best teachers and dismissal for the worst? Would you accept an error rate of 36%? 26%? How about even 12%? Would you want your employment to depend upon a measure that is accurate less than 9 times in ten, when you had been working in that job for a decade?
What if there were a well-regarded study, paid for by your tax dollars, that said shouldn't do this?
There is. recent study by Mathematica on behalf of the US Department of Education's National Center of education Evaluation and Regional Assistance, a part of the Institute or Education Sciences, offers precisely those statistics on the accuracy of value-added assessment.
Peter Z. Schocet and Hanley S. Chiang did the study, which is titled Error Rates in Measuring Teacher and School Performance Based on Student Test Score Gains (NCEE 2010-4004). They analyzed data for teachers in upper elementary grades, using a value-added approach that was based upon data of typical test score gains, class size, and number of teachers in a typical school or district. They attempted to determine who were the relatively high and low performing teachers when compared with the average teachers in the district.
With two years of data, the error rate for making such a distinction was 36%
With three years of data, the error rate was still 26%.
With ten years of data, there was still an error rate of 12%.
This is just ONE of MANY technical problems with the current status of value-added assessment models. It is one reason many of us who know anything about assessment data, even when it is massaged by value-added methods, were so infuriated by what the LA Times did. It is also why so many of us in education object to basing a substantial portion of our compensation upon student test scores, even if those scores are the results of value-added methodologies.
Remember, value-added assessment is supposed to be the magic bullet that answers questions raised about using student test data as a primary factor in evaluating teachers. It is, according to advocates for its use, the best single way of controlling for other than the effects of the individual teacher.
One of the requirements for an application for funds under Duncan's Race to the Top was tying some substantial portion of teacher compensation to student test scores. States competing for such funds in some cases had to change laws prohibiting that. I believe that all 10 of the 2nd round of winners are now legally committed to basing some significant portion of teacher compensation to a seriously flawed method of determining teacher effectiveness. Using a value-added analysis does not overcome the flaws.
All of this is preface. The information I offered above has been public for some time. For some reason editorial writers and commentators have ignored it. Just as they have ignored other evidence that demonstrates the problems with what some advocates have said are the wonders of value-added assessment or of other means of using student test scores as a method of high stakes evaluation of teachers.
On Sunday an important policy brief will be published. It is signed by ten major figures in education policy. Several have been presidents of one or more of the important professional associations that address educational measurement (the three most important are the National Council for Measurement in Education, the American Educational Research Association, and the American Psychological Association). They may serve now or have served previously as deans of schools/colleges of education that are universally recognized as leaders in educational research. Others are nationally known figures in educational policy in other ways.
The report has been in the works for several months. It ties together the available relevant research on the problems of using student data to assess teacher performance, even when value-added analysis is applied. It would have come out even without the LA Times series, but that publication makes the brief even more relevant.
I will be posting here about that brief as soon as it is legal to do so. I urge all who have any interest in this subject, whether you agree with me that using test scores in such a fashion is bad policy or advocate an approach which considers such measures of student achievement to be essential to look for that diary.
Meanwhile, feel free to comment about what I have posted in this diary.
And please, look for a diary by me that will go into far more detail, in the very early hours (just after Midnight) on Sunday morning.
Peace.