One puzzle for academics, myself included, is why businesses don’t experiment more?
Why Don’t Businesses Experiment More?
Experiments have great potential to improve business outcomes. Often businesses don’t seem to do much experimenting.
Companies pay amazing amounts of money to get answers from consultants with overdeveloped confidence in their own intuition. Managers rely on focus groups—a dozen people riffing on something they know little about—to set strategies. And yet, companies won’t experiment to find evidence of the right way forward.
Ariely, 2010, page 34
There are likely several reasons for this. Consultants give answers and answers are nice. Even if the consultant isn’t correct they give confidence. Even more critically the consultants probably support what a senior executive thinks is a good idea.
Do Experiments Discriminate Amongst Customers?
One of the more interesting objections is that business experiments often mean you aren’t treating all customers the same. Still, is this fair? Not really, I’d argue. It seems to me if it improves customer outcomes in the long-term the risk of testing is worth taking. You don’t know any customer is getting worse outcomes from the outset. After all, you only test if you don’t know (even if you have a guess). As such, I don’t think you are being unfair to any consumers. Plus consider the nature of random assignment which is key to the best tests. You aren’t deliberately discriminating against any group of consumers as they could be in the group that turns out to get the best offer.
Experimentation is used effectively in medicine to improve our knowledge. Business rarely has such important outcomes which makes the downside risks of testing much less. We, therefore, don’t need to be as careful as medicine. So let marketers do much more testing.
For more on experiments and testing see here, here, here, and here.
Read: Dan Ariely (2010) Why Businesses Don’t Experiment, Harvard Business Review, April