One puzzle for academics, myself included, is why businesses don’t experiment more. Experiments have great potential to improve business outcomes. Yet often business don’t seem to do much experimenting.
“Companies pay amazing amounts of money to get answers from consultants with overdeveloped confidence in their own intuition. Managers rely on focus groups—a dozen people riffing on something they know little about—to set strategies. And yet, companies won’t experiment to find evidence of the right way forward.” (Ariely, 2010, page 34)
There are likely several reasons for this. Consultants give answers and answers are nice. Even if the consultant isn’t correct they give confidence and probably support what a senior executive thinks is a good idea.
One of the more interesting objections is that business experiments often mean you aren’t treating all customers the same. Is this fair? It seems to me if it improves customer outcomes in the long-term the risk of testing is worth taking. You don’t know any customer is getting worse outcomes from the outset. After all you only test if you don’t know so I don’t think you are being unfair to any consumers. By the nature of random assignment which is key to the best tests — you aren’t deliberately discriminating against any group of consumers.
Experimentation is used effectively in medicine to improve our knowledge. Business rarely has such important outcomes which makes the downside risks testing much less. We don’t need to be as careful, lets do more testing.
Read: Dan Ariely (2010)Why Businesses Don’t Experiment, Harvard Business Review, April, https://hbr.org/2010/04/column-why-businesses-dont-experiment