Cathy O’Neil’s book – Weapons of Math Destruction – is an entertaining and informative read. She has done a great job of highlighting the challenges with math models. (I have one massive problem with the book but I’ll detail that next time, see here). First I’ll address the question on everyone’s mind: Is Math Racist?
Math Models Can Codfy Inequity
A summary of the book might be that math models can codify inequity. The classic example is the use of postcodes in prediction models. On the face of it throwing zip codes into a predictive model seems reasonable. It is likely to improve the prediction. After all it isn’t explicitly biased against anyone. The problem that O’Neil correctly highlights is that such inputs to models can cause unfair outcomes even if no one deliberately aims for the model to do this. Zip code is often highly correlated with race. By using zip code you end up using race as an input by proxy.
If you use models that predict a job applicant’s work success you will give jobs to those who appear, in model terms, to be similar to those who have been successful in the past. Clearly if all your senior executives are white men you shouldn’t expect this to change if you start using a model that selects for “similar” people.
Models Can Be Opaque
Compounding the problem is that many people using them don’t understand the models. They therefore can’t really critique them or compensate for the model’s problems.
To O’Neil Weapons of Math Destruction cause major problems — they are the dark side of the big data revolution. She suggests we worry when math models are
“..opaque, unquestioned, and unaccountable, and they operate at a scale to sort, target, or “optimize” millions of people. By confusing their findings with on-the-ground reality, most of them create pernicious WMD feedback loops.”
O’Neil, 2016, page 12
The last point is especially interesting. Models can help craft reality. Police see areas as likely to have high crime, they patrol more, see more crimes, and so the area becomes seen as even more likely to have high crime.
Is Math Racist?
There are great points in O’Neil’s book and even those who love big data should read it and think through the problems raised. Math (models) can be racist if not managed properly. We shouldn’t accept a model, however sophisticated, without giving very serious thought to its inputs and consequences.
Read: Cathy O’Neil (2016) Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, Crown, New York.