Skip to main content

View Diary: Atlanta cheating scandal highlights dangers of high-stakes testing (146 comments)

Comment Preferences

  •  May I cite another example from a different (9+ / 0-)

    context? Before I retired from the government and became a teacher, then a professor, I was an IT manager at a government agency. The higher-ups in my chain-of-command decided that they needed "metrics" to monitor our customer service efforts. So the main metric they chose was a count of the number of "trouble tickets" technicians closed per month. Advocates of this metric claimed that this would "hold technicians accountable" for fixing customer problems promptly.

    Here's what really happened: Knowing that's how they would be judged, technicians simply started closing more trouble tickets, even if they hadn't fixed the problems the customers phoned in (or had only partially fixed them). They knew that customers would simply phone in new reports if the problems bothered them enough, but in the meantime they (the techs) got credited with "closing the tickets".

    End result: customers became far more dissatisfied, because it seemed their reports were being ignored or not treated seriously. Ending to the story? Ten years later, the same problem persists, because the management still refuses to believe its metric is flawed.

    I believe that in every country the people themselves are more peaceably and liberally inclined than their governments. -- Franklin D. Roosevelt

    by Blue Knight on Thu Jul 07, 2011 at 10:37:18 AM PDT

    [ Parent ]

    •  The type of problem you cite is real... (2+ / 0-)
      Recommended by:
      Odysseus, drmah

      ...and I've seen it in private industry as well. BUT, when it's happened in private industry, I've not seen it last very long (less than 12 months in most cases as simple to game as this) when the measured end results don't improve as projected.

      The fact that some metrics are inappropriate does not mean that all metrics are inappropriate. In the case you describe, random auditing of the claimed resolution with serious consequences (such as deducting 50 closed tickets for each improperly closed one and increasing the chances of that tech's tickets being audited) for closing a ticket as "fixed' when it was not might have gone a long way to making this metric more useful at incenting the desired behavior.

      This is similar to the problem sales departments have. If the salespeople are not selling the desired mix of products, it's often because the commission structure is just wrong (although, sometimes, product quality, training, availability, etc are a significant factor as well). Salespeople are programmed by their incentive structure nearly as surely as a CPU is programmed by the sequence of instructions presented to it.

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site