Cathy O’Neil has a great book on big data but one with a fundamental flaw. The flawed claim is made in the book’s subtitle and permeates throughout the book. The subtitle is: “How Big Data Increases Inequality and Threatens Democracy”. I could find no significant evidence of big data increasing inequality in the book. This is not to say that big data couldn’t, or doesn’t, increase inequality, but O’Neil makes no serious attempt to show that this happens never mind how it happens.
This is a massive problem. It means O’Neil cannot make meaningful recommendations. Should we destroy all models? I have no idea because she doesn’t show if the machines are making life better, worse, or having no net impact.
To my mind O’Neil’s is an example of the sloppy thinking that permeates politics. The right wants to make countries great — implying that there has been a decline but from when exactly is never specified. Some on the left, who might be expected to believe in progress, instead accept this characterization of decline. Little evidence is ever offered, only a vague feeling that things were better in the past. (Do these people not watch Mad Men?)
To be clear O’Neil’s book contains numerous examples of significant problems in big data models. But O’Neil’s claim about increasing inequality is unsupported because pointing to evidence of a problem now is not the same as pointing to evidence of an increasing problem.
Strangely the problem with her logic is evident just from listening to what she says. She clearly knows that bias didn’t get created along with the first PC. She describes how housing loans pre-big-data-models were awarded in a racist fashion. She mentions that people exhibit bias when they make decisions without using big data models. She even says that “..racists don’t spend a lot of time hunting down reliable data to train their twisted models” (O’Neil, 2016, page 23). Unfair bias has been with us probably as long as people have existed.
The obvious unanswered question is: have math models made things worse? Policing and the judicial systems have had, and still have, problems with being unfair but are they worse? To do this O’Neil must specify a baseline — how biased decisions were before the adoption of the models she complains about and compare this to the results after the models. To labour the point if loan and policing decisions were racist before the adoption of math models then documenting evidence of racism after the adoption of the models isn’t enough to show they are more racist now. We need to know whether there is more or less racism now.
O’Neil has some great points yet the error she makes is pretty basic — it is intellectually sloppy to claim things are getting worse without providing any supporting evidence.
As we see an end to 2016 I think it is important that people who believe in progress don’t accept that society is inevitably plunging towards doom. O’Neil has an important point — math models can codify bias — but the models could also help make the world a better place. Crucially, we need to test when, and how, progress can be made and not just assume that the world is falling apart. Such knee jerk negativity only helps those who don’t believe in progress.
Read: Cathy O’Neil (2016) Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, Crown, New York.