Cathy O’Neil’s book – Weapons of Math Destruction – is an entertaining and informative read. She has done a great job of highlighting the challenges with math models. (I have one massive problem with the book but I’ll detail that next time).
A summary of the book might be that math models can codify inequity. The classic example is the use of postcodes in prediction models. On the face of it throwing zip codes into a predictive model seems reasonable. It is likely to improve the prediction and isn’t explicitly biased against anyone. The problem that O’Neil correctly highlights is that such inputs to models can cause unfair outcomes even if no one deliberately aims for the model to do this. Zip code is often highly correlated with race and so by using zip code you end up using race as an input by proxy.
If you use models that predict a job applicant’s work success you will give jobs to those who appear, in model terms, to be similar to those who have been successful in the past. Clearly if all your senior executives are white men you shouldn’t expect this to change using a model that selects for “similar” people.
Compounding the problem is that many people using them don’t understand the models so can’t really critique them or compensate for the model’s problems.
To O’Neil Weapons of Math Destruction cause major problems — they are the dark side of the big data revolution. She suggests we worry when math models are: “..opaque, unquestioned, and unaccountable, and they operate at a scale to sort, target, or “optimize” millions of people. By confusing their findings with on-the-ground reality, most of them create pernicious WMD feedback loops.” (O’Neil, 2016, page 12). The last point is especially interesting. Models can help craft reality. Police see areas as likely to have high crime, they patrol more, see more crimes, and so the area becomes seen as even more likely to have high crime.
There are great points in O’Neil’s book and even those who love big data should read it and think through the problems raised. We shouldn’t accept a model, however sophisticated, without giving very serious thought to its inputs and consequences.
Read: Cathy O’Neil (2016) Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, Crown, New York.