Regression to the Mean

Daniel Kahneman likes to explain an epiphany he had teaching decision making. (The story is so good you realize you’ll never come up with as perfect an illustration, but then comparing yourself to a Nobel prize winner is never good for the ego). Kahneman was instructing the Israeli Air Force. He announced that rewarding good performance is more effective than punishing failure. The military people thought this was wimpish academic nonsense. An instructor told him that whenever cadets were praised they did worse, and whenever shouted at they improved. The instructor had evidence, but Kahneman knew about Regression to the Mean.

Regression to the mean happens because most things in life have a deterministic, and a random element. Skill (deterministic) matters, but so does (random) luck. When you see a fantastic sporting performance chances are that it is a talented person having a lucky day. Talent is fairly stable over time, but luck isn’t. After doing something exceptional you probably won’t have that much good luck again next time. When failing abysmally you probably had some bad luck, and will likely have better luck next time. An exceptional performance, on average, is followed by a worse performance and a terrible performance, on average, is followed by a better one.

“Naturally, he praised only a cadet whose performance was far better than average. But the cadet was probably just lucky on that particular attempt and therefore likely to deteriorate regardless of whether or not he was praised. Similarly, the instructor would shout into a cadet’s earphones only when the cadet’s performance was unusually bad and therefore likely to improve regardless of what the instructor did. The instructor had attached a causal interpretation to the inevitable fluctuations of the random process.” (Kahneman 2011, page 176)

Regression to the mean explains numerous phenomenon. A very tall mother is likely to have a taller than average daughter, but probably a daughter shorter than she is. The mother’s “tall genes” added to “tall luck” made her very tall. The child may inherit the “tall genes” but not the luck. The random element is unsurpisingly, on average, likely to be average.

The instructor noticed praise leads to worse performance but the lesson is that unless we understand why things happen we can’t really hope to influence the world.

This also illustrates that fantastic examples can be found in many places. Please let me know if you see any.

Kahneman’s book is well worth a read. It is definitely psychology, not behavioral economics, see Kahneman’s Gripe, but none the worse for that.

Read: Daniel Kahneman, Thinking Fast and Slow, 2011, Doubleday Canada