In business decisions are often taken “without having any real evidence to back them up” (Davenport, 2009, page 69). This is a source of great frustration to me, (and many academics). To be fair sometimes there isn’t really any choice. Davenport explains that this is often the case for big strategy decisions. One simply can’t test some massive changes to the way the firm operates. Tactical changes are often much more amenable to testing. As he says, “[g]enerally speaking the triumphs of testing occur in strategy execution, not strategy formulation” (Davenport, 2009, page 71).
Given its strengths in tactical decisions it is no coincidence that testing is big in the field of credit card offerings. One can offer one set of customers a new offer, another set of customers the traditional offer and, provided the customer groups were about the same going in, we can ascribe any differences to the offers made. Davenport highlights how we set up a control; in the example above the control group is a set of customers similar to the customers receiving the new offer. Random assignment between the test and control conditions, which is relatively easy with lists of customers/online marketing, should ensure that the two groups are pretty similar before they receive the offers.
The critical thing to adopting a scientific approach is that the tests have to be real tests. Davenport notes that “tests” are often run that aren’t really rigorous. This is especially worrying if they are described in scientific terminology. One might think the terminology used, laboratories etc.., is suggestive of proper testing when none of the benefits of a proper test are received.
Davenport offers six +1 easy steps to running a test and in his article explains what happens at each stage.
- create or refine hypothesis
- design test
- execute test
- analyze test
- plan rollout
The plus one step is important — create a Learning Library. This will capture the insights for posterity. There is little point in gaining knowledge from tests if everyone just forgets about it as soon as it is finished running.
Thomas Davenport has quite a knack of explaining ideas clearly. Anyone planning to try testing at their firm would do a lot worse than look at his advice.
Read: Thomas H. Davenport (2009) Smart Business Experiments, Harvard Business Review, February.