This diary will be -- as are most of mine -- about education. It will be slightly different. It is about the distortions of testing.
A friend of ours, Jeanne Heifetz, became quite well known a while back when she found out the tests her child was taken had altered passages of literature. She raised a stink about it, and in theory NY State promised not to do that again.
(for some imformation on Jeanne, you can try here, or this or simply go out and google her.)
The story below the fold will point at another problem.
I will offer the complete story, from the
Houston Chronicle, and then make my comments afterwards.
Author used in TAKS flunks test
By RICK CASEY
Copyright 2005 Houston Chronicle
If any of you students who took the TAKS test this week get your reading scores and find that you were marked wrong on a number of the answers, here is what you do:
Call the author of the piece on which you were tested and ask him or her to intervene for you.
While you were hunkered down this past week taking the test, I was talking to one of the authors whose work appeared in last year's 10th-grade test.
Naomi Shihab Nye is one of Texas' best poets and a fine essayist and novelist.
She was a finalist for the National Book Award two years ago, has four times won the Pushcart Prize for small press literature, has appeared on two PBS poetry specials (one with Bill Moyers) and on A Prairie Home Companion.
She is a woman of great warmth and wisdom, qualities that make her a frequent "writer in residence" in schools.
She is also a good friend and former neighbor, so I was eager to hear how she felt about having one of her essays used in the TAKS test.
Her first response was that she was honored to have one of her essays chosen, and that it actually helped her work with a group of students at George West High School in South Texas last spring.
She had forgotten that she had, many months earlier, given permission for the essay to be used in the test. Now she was with students who had just taken that test.
'One humorless fireman'
"When I walked into the George West library, the students' feelings of friendliness and warmth really hit me," she said. "It felt like they already knew me.
They already had things they wanted to talk about."
The essay begins with her son Madison, then 4, writing out her name and his and drawing lines from the letters in his name to the same letters in hers.
"Naomi, look, we're inside one another, did you know that? Your name is here, inside mine!"
She muses, in her vivid, poetic fashion, on her name, which means "pleasant," and on the changes in her life, including falling in love, marrying and taking on
another name.
It ends with her and her husband inviting every Nye in the San Antonio phonebook to a potluck dinner at their house. All but "one humorless fireman" showed up.
She said the students seemed energized by the story to tell stories of their own, which is exactly what she wanted.
Details stick in the minds
On her way home from George West, Naomi was struck by how the students had remembered details of the piece but, when she asked, could remember none of the questions. When I sent her copies of the questions, she said "It reminded me of the trouble I always had with standardized tests."
The trouble?
"Almost every question has more than one 'right' answer," she said.
That's the difference between testers and writers. Poets and other literary writers see literature as a collaborative engagement between the writer and the reader. They expect different readers to have different reactions to their work, to draw different messages based on their experiences and concerns.
So she had a problem with a question that asked what the essay was "mainly about." Answers included "moving to a new place" and "the significance of names."
"Say a kid had just moved to a new place and had a lot of revelations about himself, that would be the right answer for him," she said. "Another kid who felt special about names would focus on that."
She had similar arguments with the answers to several other questions, arguments she had had since, at age 22, one of her poems was selected for a textbook.
"Out of five questions the kids were supposed to answer, I couldn't answer three," she said.
Literature is about exploring, not about measuring.
Naomi has seen "a growing desperation" among teachers with whom she works in the schools as they are pressured to focus on the test.
Teachers with the skills and the circumstances to lead their students in creative exercises find them doing better on the standardized tests. Teachers mired in almost constant test preparation worry about them doing worse in life.
You can write to Rick Casey at P.O. Box 4260, Houston, TX 77210, or e-mail him at
rick.casey@chron.com.
Now as promised, my comments.
Our use of selected response [that is education-speak for mulitple choice] items [question] is that they are binary [one answer correct, evertying else wrong] an encourage convergent thinking, not divergent thinking. There provide little room for creativity, or for explaining how the student arrived at an answer, nor do they provide partial credit for say a second best answer.
I would argue that advances in human knowledge do not for the most part come from recognizing pre-determined "right" answers. In fact, advances in knowledge come far more from being able to frame questions. That would include being able to examine one's own reasoning processes -- a process one can describe as meta-reasoning. Clearly most advances in science and technology come about because people arrive at new ways of thinking about problems.
We as teachers in fairness must prepare our students for tests such as this. Having done SAT work [for Princeton Review and for a smaller company] I know how easy it is to prep for selected-rseponse items. You can inflate your score by not looking for the correct answer, but by eliminating obviously wrong answers [even on SAT, where there is a correction for guessing, which there is not on most state tests where all that matters in the # of questions answered correctly]. Follow this scenario
- each question has four answers
- there are 30 questions
- you actually know the correct answer on 10 questions
- you can eliminate two answers on each of the 20 left
A true measurement of your score would be 10/30, or 33 1/3%, and clearly you would fail.
But on average, you will, with only 2 answer left, get 10 of the remaining 20 correct. Thus your score will be 20/30, or 66 2/3% , and you will pass -- with a test score that is twice as good as you really know
Now before anyone accuses me of manipulating test results, I'd like to point out that by approaching the test this way the student gets some reward for knowing the second best answer -- that is, s/he is gaining partial credit for knowing what answers are wrong.
I also teach my stronger students to use this technique to figure out -- in cases of more than one correct answer or no technically correct answer -- to think as the test writer, and to determine the answer they want, even if it is either not the best answer or not even actually correct. An example of not correct would be a question that asks which Supreme Court Decision overturned Plessy v Ferguson and includes in its answers Brown v Board. It's obvious which answer will be marked correct even though it si wrong, because Brown was decided on the basis that segregated schools were inherently unequal and hence could not meet the test [separate but equal] established in Plessy. And I have seen precisely this question on a mandatory state test.
The very first test I ever made up, when I was a teacher inern in a Quaker secondary school back in 1974, more than 20 years before I actually became a teacher, had two essays. It was for a course in "Great Issues in American History" for juniors and seniors. For each essay the students were to pick one of a pair of quotes, say who among the people we had studied had written it, and why.
In each pair one quote was from someone we had studied, albeit it was not a passage we had studied. if the student recognized the writer, they might do okay from that recognition. The other quote was NOT from any of the people we had studied. How then could I grade their answers, especially if they took the "other "quote."
Remember the qustions requires an explanation of "why" the person chosen would have said the quote. he entire grade was based on the coherence and support of that "why." If the student could make a good case, including where possible reference to things that persond DID say, they fully answered the question and were so graded.
Now I ask you, who is demonstrating greater understanding, a student in that Quaker school who picks the wrong person but gives a very thorough and supportable answer, of one of my weaker students who passes a multiple choice test because s/he can eliminate wrong answers and can thereby pass?
I will acknowledge that the creation of tests and other assessments that are reliable [give consistent results] and valid [that is, the information we derive from tham allows us to draw valid inferences] is not the easiest task in the world. Many teachers struggle with it, even those of us who are presumed to be strong teachers. That is why we have the unfortunate situation where textbook vendors now provide entire kits -- prewritten test questions in a pool from which you generate your test.
For me, before I begin to teach a topic, I have to have some idea of how i cam going to assess the learning of my students. It has to be something that provides meaningful feedback both to them and to me. I do not make up the tests ahead of time, because I do not know what will actually occur during the time we explore that material. And I may find that I cannot ask the questions in exactly the same way in classes of equal level students that I teach back to back, because what happened while were studying that material varied greatly bewteen the two. I must assess in a manner that will accurately measure what happened IN THAT CLASS WITH THOSE STUDENTS
Selected response qustions are easier to generate than good constructed response questions. They are clearly less expensive to administer and grade [by machine] than are constructed response questions. Our increasing dependence upon them is in fact impoverishing the thinking of our students. It is possible to write selected response questions which invoke higher order thinking, but from ahving taught in two states, i will tell you that encountering such on these high stakes tests is quite rare. Far too much in the case of history and government becomes little more than recall, That bespeaks a model of education where the teacher pries up the top of the students' heads and pours in the knowledge. It indicates a mentaily that does not question received knowledge, and thus may very well match the mentality of many in power today -perhaps they do not want to develop a generation of students capable of higher level thinking, of synthesis, of asking probing questions.
I am a teacher. Clearly that is one reason why I keep returning to these issues on what is a blog primarily about political and social issues. I strongly believe this both -- political and social. How we educate our students will play a major role in shaping our future as a nationand a people. The way we assess that learning will inevitably shape not only what we learn, but how we learn. And that learning process will be, far more than the specific facts on which students are tested, what students will carry with them as they go on in life.
I won't post a mojo mug, nor ask for recommendations. I will leave it up to the readers how much visibility is received on this diary.
Have a nice Sunday.