Weapons of Math Destruction is a well written, important book about how we are all basically screwed. I feel like I am writing that a lot lately, and I don’t think that the author, Cathy O’Neill, would agree with my assessment, but we will get that in a moment. In the meantime, the book itself is a most read. It is the best work, in my opinion, on where we are today and where we are likely to end up, with the respect to the power that data and algorithms have over our daily lives.
O’Neill is a fantastic story teller. Each of the book sections is written almost like a thriller. A wrong has been committed – to a teacher, or a retail employee or a minority mortgage seeker – and O’Neill methodically traces how an algorithm is to blame for the harm. Step by steps she takes the reader through what makes a good mathematical model and demonstrates how the model and algorithms at fault fail to adhere to those standards. She also demonstrates how the owners of those models use them to make end runs around the laws meant to protect us from unfair, discriminatory and predatory actors.
In all walks of life, O’Neill shows how companies and other private interests have substituted their own private algorithms in an attempt to get around rules designed to prevent them from unfairly discriminating against people. Companies use zip code as a proxy for race, for example, or recidivism predicting models fail to account for racial bias in arrests and sentencing. Racism and discrimination are therefore perpetuated but laws are arguably not violated. Companies leverage their economic power to force “schedule” efficiencies on powerless workers that mean employees don’t know more than a few hours when they have to be to work, making planning school or childcare impossible. And all of this happens on the edges of what is legal and with no oversight or feedback to ensure that the models improve or are accurate.
Not all models are bad, as O’Neill points out. Models that are based on reasonable assumptions, strike the right balance between the harms of a false positive and the harm of a false negative, have appropriate feedback mechanisms to ensure improvements in accuracies, and are open to having all of the above elements inspected can and have done great good for the world. But the vast, vast majority of algorithms in private hands follow none of the rules.
And you might think that is why we are screwed. But that is the symptom, not the disease. The disease is this:
That’s not to say that algorithms should be universally outlawed or forced to be open sourced, but it does mean that the burden of proof rests on companies, which should be required to audit their algorithms regularly for legality, fairness, and accuracy.
If one of the leading lights of the fight against weapons of math destruction is unwilling to attack the root of the system, then we are screwed. If we are to deal with this issue, and we must, then we need to deal with it at its root. We need to control their access to our data, force them to open their models to public inspection, and put strict limits on where models can and cannot be used. We are on the cusp of something new and powerful in ways we have not dealt with before. It has the potential to be very dangerous. Half measures and the guardrails developed for industrial capitalism aren’t going to be enough. The power these models can have over society is a difference of kind form anything we have seen in the past. It demands a level of societal control different than anything we have impose din the past.
Read this book. It is extremely well written by perhaps the most intelligent observer of the new world of algorithmic decision making we have (her blog mathbabe.org is also a must read). It explains what is and is not a good model and how bad models are slowly eroding our society. It will help you understand the new world we are making and how to understand when a model is good and when it is bad. Just don’t accept that her solutions are acceptable, sufficient or all that can be done or should be done in the face of this brave new math.