Three major names in the field of decision-making teamed up to write about the problem of noise. Daniel Kahneman, who won the Nobel prize in economics despite not being an economist, Oliver Sibony, apparently a knight in the French Order of the Légion d’Honneur, and Cass Sunstein of Barack Obama’s White House and Nudge fame, make an impressive team. The book is, unsurprisingly, a good one.
What Is Noise?
The authors were writing a popular book (although informed by science). As such, they want to avoid terms like variance and, especially, standard deviation. (While variance isn’t too bad, the latter term feels like it was explictly designed to put people off). They settle on noise as their hook. Noise is unwanted variation in judgment. The authors spend a decent amount of time explaining that the unwanted bit matters. The fact that two people convicted of the same offense get different sentences is fine, if they really represent different cases. What makes the difference unwanted is when, for example, people deserve the same sentence but one gets a light sentence and another a harsh one because of luck. One got a lenient judge, the other a hanger. Or one person got sentenced when the judge was tired. This shouldn’t happen. We should be annoyed by this.
The Problem Of Noise
The authors are keen to highlight that noise can be a problem in a large range of situations. They talk a lot about the legal system. And that makes sense. Indeed, one can think of the development of laws as a way to reduce noise. Early codification of legal systems — Hammurabi’s, Draco’s, Solon’s — can be seen as a way to try and ensure that those who perform the same offenses get similar punishments. Some punishments in ancient codes were pretty brutal. I can remember teachers trying to tell us how a written law might seem harsh to us, draconian even. Yet, Draco’s code was fairer than the random local big wig inventing a punishment. Given there was a pretty good chance you ended being executed for any crime in the ancient world, it took a lot of effort to see ancient lawgivers as their age’s liberal heroes.
The problem of noise figures in other situations. In business, they note that those providing a quote for insurance should ideally be noiseless. Someone who gives randomly high quotes loses business, while randomly low quotes will lose the firm money on the business it gains.
I bought their argument that there is a decent amount of situations where noise is a problem. The fact that errors cancel out on average doesn’t help the person serving an unnecessarily long sentence even if they have the consolation that someone else was let off with a slap on the wrist.
The large role of noise in error contradicts a commonly held belief that random errors do not matter, because they “cancel out”. This belief is wrong.
Kahneman, Sibony, and Sunstein, 2021
Spotting Noise
Here it is important to understand the difference between bias and noise which they helpfully illustrate with a target metaphor. Bias is a predictable skew away from the target. Thus bias in a judicial system occurs when irrelevant characteristics, e.g., race, have a predictable effect on outcomes, e.g., sentences. This is, of course, bad.
The authors argue we are more likely to miss another problem, noise. Errors that are not wrong on average but create problems because the individual outcomes, not just the aggregate, really do matter. How to spot noise then?
A general property of noise is that you can recognize and measure it while knowing nothing about the target or bias.
Kahneman, Sibony, and Sunstein, 2021
An important point is that you can spot noise if you know there is a right answer even without knowing what the right answer is. If you see marks all over the target you know that there were a lot of errors even if you don’t know what part of the target people were actually aiming for. The mere presence of the variability shows evidence of error.
Tackling Noise
They suggest a noise audit to find noise and decision hygiene rules to improve judgments. They helpfully layout how to do these. Their advice makes sense to me.
For related posts see here, here, here, and here. And for my page on public policy and decision-making see here.
Read: Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein (2021) Noise: A Flaw in Human Judgment, Little, Brown and Company