Cathy O’Neil has a great book on big data, Weapons of Math Destruction, but one with a fundamental flaw. The flawed claim is made in the book’s subtitle and permeates throughout the book. The subtitle is: “How Big Data Increases Inequality and Threatens Democracy”. I could find no significant evidence of big data increasing inequality in the book. She shows a problem, but is it worsening? There is no meaningful attempt to show this.
This is not to say that big data couldn’t, or doesn’t, increase inequality. Still, O’Neil makes no serious attempt to show that this happens, never mind how it happens.
Recommendations May Differ Depending Upon Whether The Problem Is Worsening
Not showing that big data increases inequality is a massive problem. It means O’Neil cannot make meaningful recommendations. Should we destroy all math models? I have no idea because she doesn’t show if the machines are making life better, worse, or having no net impact.
To my mind O’Neil’s is an example of the sloppy thinking that permeates politics. The right wants to make countries great again — implying that there has been a decline. From when exactly is never specified. (Am I the only one who thinks the 50s wasn’t peak civilization?)
They have suprising allies on the left. Progessives might be expected to believe in progress, but they often don’t. Instead they often accept this casual characterization of decline. Little evidence is ever offered, only a vague feeling that things were better in the past. (Again did these people not watch Mad Men?)
She Shows A Problem But Is It Worsening?
To be clear O’Neil’s book contains numerous examples of significant problems in big data models. But O’Neil’s claim about increasing inequality is unsupported because pointing to evidence of a problem now is not the same as pointing to evidence of an increasing problem.
Strangely the problem with her logic is evident just from listening to what she says. She clearly knows that bias didn’t get created along with the first PC. She describes how housing loans pre-big-data-models were awarded in a racist fashion. O’Neil mentions that people exhibit bias when they make decisions without using big data models. She even says that “..racists don’t spend a lot of time hunting down reliable data to train their twisted models” (O’Neil, 2016, page 23). Unfair bias has been with us probably as long as people have existed.
Have Math Models Made Things Worse, Better, Or Has It Stayed The Same?
The obvious unanswered question is: have math models made things worse? Policing and the judicial systems have had, and still have, problems with being unfair. Still are they worse?
To show a decline O’Neil must specify a baseline — how biased decisions were before the adoption of the models she complains about. She should then compare this to the results after the models were introduced. To labor the point if loan and policing decisions were racist before the adoption of math models then documenting evidence of racism after the adoption of the models isn’t enough to show they are more racist now. We need to know whether there is more or less racism now.
O’Neil has some great points yet the error she makes is pretty basic. It is intellectually sloppy to claim things are getting worse without providing any supporting evidence of this fact.
We Aren’t (Necessarily) Plunging Towards Our Doom. (We Need Evidence To Show This)
I think it is important that people who believe in progress don’t accept that society is inevitably plunging towards doom. (I wrote this in late 2016 when it did seem like doom was especially near). For a more positive view see here and here.
O’Neil has an important point — math models can codify bias — but the models could also help make the world a better place. Crucially, we need to test when, and how, progress can be made and not just assume that the world is falling apart. Such knee jerk negativity only helps those who don’t believe in progress.
Read: Cathy O’Neil (2016) Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, Crown, New York.
