is the title of a very interesting post from England by David Boyle that has made the rounds (partly with my assistance) of some education lists and sites.
After all, one of the real problems in education is our national obsession with measuring, as if weighing a pig 5 times a day would actually increase its weight.
Let me offer two "rules" from the piece and then discuss the implications:
Goodhart’s Law (after former Bank of England director Charles Goodhart) which says that: any measure used to control people – and all targets are that – are bound to be inaccurate. No matter how inefficient an organisation is, their staff and managers can always manipulate the definitions so that their target numbers look better.
Never mind Goodhart’s Law, which is about using numbers for control. Here is what I humbly submit as Boyle’s Law: When you use numbers as the basis for payment, they become irrelevant to the broader objectives of the service.
Let me offer a few more selections from Boyle to help set the context in which he writes (although of course you could simply read his piece and dispense with my efforts):
Measuring by ‘outcomes’ – measuring the effects of the activity rather than the activity itself – is in.
Measuring things has gripped the zeitgeist by the throat because numbers seem to provide a hard-headed anchor of certainty in a relative world.
The problem isn’t with the numbers themselves, it is with the definitions at the other end. Those tough-minded mandarins who demand robust numbers forget that, chained to the numbers are some very un-robust, rather fluffy and subjective words, which can be endlessly manipulated.
Of course, at least in education the tendency is to measure what is easy to measure, even if it is not important. Although Boyle does have a gift for making that point: some very un-robust, rather fluffy and subjective words, which can be endlessly manipulated. Oh yes.
In education, we obsess about scores that are "scaled" from "raw scores" that may be a poor sampling of what students can do or actually know. But never mind.
I am of an age. . . that is, I was born in 1946, so I was of a critical age during Vietnam. I was in the military (Marines) but only stateside. And I remember our obsession with the number of enemy killed - although at least then we did not kill the #3 in Al Qaeda multiple times. Body Counts. That was the metric. Or if not that, then tons of bombs delivered on targets. Or some other thing we measured merely because we could. But what was "on target?" And who were the "enemy dead?" Those were valid questions then, and they certainly are now, although perhaps our phrasing is a bit more "artful" leaving just a wee bit of "wiggle room" for error - "suspected terrorists." Of course, were we in a court of law we would not be allowed to execute someone merely because they were "suspected" of anything. But don't worry. "We" control the definitions, and things are what we say they are. "Suspected" is close enough for military work, eh? And those women and children who killed at the same time? Why not stretch the the language further while we are at it and call them future suspected terrorists, or breeders of suspected terrorists? Heck, we need not be so specific - "unavoidable collateral damage" does just fine - and can we please get back to the numbers, the hard data?
Remember what I cited above, from the author, Boyle’s Law: When you use numbers as the basis for payment, they become irrelevant to the broader objectives of the service. Here's what Boyle offers immediately after stating his law:
This will, in short, hollow out and subvert the service that is being paid for, just as bonuses – also a kind of payment by results – did for banking. It will shift the energy and focus – not to encouraging services which can genuinely prevent – but to selecting the clients who can fulfil the definitions of the outcome with the minimum effort. Worse, it means excluding clients who are difficult or even just difficult to define.
Let me translate this to teaching. We have in education the term "bubble kids." Those are the ones right around the cut scores. What this points at is that with the emphasis on test scores, the temptation is to ignore those who are bright because they will pass anyhow, get rid of the hard to educate like Special Ed or English Language Learners, because they probably won't pass, and focus on the kids right below the cut score, then brag about how much you have improved scores, pocket the bonus, then - if you are smart - move on to something else before people get wise.
Or if you control the translation of raw scores to the scaled scores, there doesn't even have to be any difference in performance - just translate the more recent scores to imply a higher performance and pocket the bonus.
That's how we get mortgages issued to people who should not qualify - but the person clearing the mortgage gets credit for the deal and doesn't have to worry about the failure of the loan later. Or we get short-term minuplation of corporate performance to juice up the stock's price so the executives can cash in their quarterly bonuses, maximize the value of their stock options, and never mind that the future of the corporation is now assured - as a failure.
Perhaps we should remember the aphorism attributed to Mark Twain, that "There are three kinds of lies: lies, damned lies, and statistics" and be cautious of the weight we give any set of numbers before we understand how those numbers are derived, to what manipulations they can be subjected, and only then can we determine what if any meaning they offer us.
This is NOT just a problem in assessing student learning, teacher or school effectiveness. Would that it were. We are seeing it there, but only after the damage it has done to our economic system, and is now doing to our governance as well.
We obsessively focus on numbers, as if somehow the fact that we can claim to quantify something gives it more meaning than if we describe it, including acknowledging the limits of our ability to fully describe.
Attempting to drive everything by numbers is to put it mildly not merely obsessive - it is insane. It sets up a paradigm that is closed to the observation of the small child that the emperor is naked - but we have the numbers!!! It forecloses the possibility of recognizing that even our mnost accurate measurements may be missing the big picture - we can account for the tire pressure, the oil pressure, the gas consumption, the rate of acceleration and deceleration, and entirely forget to look up and notice we are on the wrong side of the road about to be hit head on by an eighteen wheeler.
We can drive more efficiently, using less gas, driving more quickly and directly towards our objective, and be oblivious to the fact that directly ahead of us is a large cliff off of which we are about to drive.
Off a cliff. With our economy. With our educational system. With our government.
But don't worry. We've got numbers on that too, don't we?
Ah, Boyle puts it right. There are some real perils with this kind of obsession.
Except in this country we seem to be obsessive gamblers. We have been proven wrong time and time again by pursuing such a path - in business, in banking, in military endeavors, in education. So what do we do? Like any compulsive gambler we double down.
Tomorrow is our last day before Winter Break. Perhaps that is why I am so cynical? I will not have the hopeful faces of my young people for a period of 11 days - at least, this year I hope it is only 11 days, and that we don't get the time extended without planning because of massive snow falls.
So does any of this make any sense? Or is this just the bloviations of a surly teacher who does not want to be held accountable? That's the accusation some will offer in my direction.
What about you?