Two stories about the urge to quantify student "progress" made headlines this week. Taken separately, they are a source of rich debate for anyone interested in education in general, and those who care about teachers, in particular. Taken together, they should be utterly terrifying for anyone who cares about either teachers or education.
A committee of state lawmakers approved changes to two bills that would create Michigan’s first statewide teacher evaluation system, delaying a final vote until next week.Please read below the fold for more on this story.
The House education committee met Wednesday to discuss substituted changes to HB 5223 and HB 5224, designed to implement a new teacher evaluation system that for the first time would include student growth in rating a teacher's performance.
And how shall we measure student growth? Teachers in Michigan had better hope that said measurements don't include something along these lines:
“According to professor of theory of knowledge Leon Trotsky, privacy is the most fundamental report of humankind. Radiation on advocates to an orator transmits gamma rays of parsimony to implode.’’The notion of computer-assessed high-stakes exams is just the latest (and, arguably, most frightening) in a string of efforts to take something (education) that is very difficult to quantify well, and place a numerical value on it.
Any native speaker over age 5 knows that the preceding sentences are incoherent babble. But a computer essay grader, like the one Massachusetts may use as part of its new public school tests, thinks it is exceptionally good prose.
Robo-graders do not score by understanding meaning but almost solely by use of gross measures, especially length and the presence of pretentious language.
Recently, three computer science students, Damien Jiang and Louis Sobel from MIT and Milo Beckman from Harvard, demonstrated that these machines are not measuring human communication. They have been able to develop a computer application that generates gibberish that one of the major robo-graders, IntelliMetric, has consistently scored above the 90th percentile overall. In fact, IntelliMetric scored most of the incoherent essays they generated as “advanced” in focus and meaning as well as in language use and style.
The concept of pushing untested (or under-tested) quantification efforts, all in the name of "accountability," are nothing new under the sun. In fact, my first foray into the subject here at Daily Kos was nearly four years ago.
What is particularly maddening about this particular episode is the fact that the research into these assessment "tools" is largely being kept under lock and key, despite the fact that many of our nation's students are being used as test subjects in "pilot programs."
As the recently retired director of MIT's writing program, Les Perelman, noted:
Unfortunately, the problem in evaluating these machines is the lack of transparency on the part of the private vendors and the researchers associated with them. None of the major testing companies allow easy access or open-ended demonstrations of their robo-graders. My requests to the testing companies to examine their products have largely gone unanswered. A Pearson vice president explained that I was denied access to test the product now being considered by PARCC because I wanted “to show why it doesn’t work."(The researchers who worked on ways to "trick" another robo-grading program actually bought their own version of the assessment tool in order to test it, by the way.)
So, to review: (1) Assessment tools are being readied for use to judge our schools and their teachers that are not able to be independently tested for their accuracy. (2) There is a strong likelihood that not one, but many, states will include those assessments to "hold teachers accountable," perhaps impacting their pay and their very employment. (3) At least some evidence exists that the machines could be completely unreliable as a legitimate vehicle of assessment.
But, it's based in technology, and it is right in the wheelhouse of the "reform" movement. And it puts a number on the "value" of teachers, which has been the chief ambition, if not obsession, of the right (and far too many on the left and center) for years. So, despite the obvious red flags, expect this to become standard policy in state after state. After all, the mantra of "educational reform," for years, has been summed up in three simple words: Ready. Fire. Aim.