We should be conducting business tests to have more confidence we are doing the right thing.
Small And Big Decisions
In business decisions are often taken “without having any real evidence to back them up” (Davenport, 2009, page 69). This is a source of great frustration to me, (and many academics). To be fair sometimes there isn’t really any choice. Davenport explains that this is often the case for big strategy decisions. One simply can’t test some massive changes to the way the firm operates. Tactical changes, however, are often much more amenable to testing. As he says, “[g]enerally speaking the triumphs of testing occur in strategy execution, not strategy formulation” (Davenport, 2009, page 71).
Conducting Business Tests For Tactical Decisions
Given its strengths in tactical decisions it is no coincidence that testing is big in the field of credit card offerings. This is because the firm can offer one set of customers a new offer. The firm then offers another set of customers the traditional offer. Provided the customer groups were about the same going in, we can, therefore, ascribe any differences to the offers made.
Davenport highlights how we set up a control. In the example above the control group is a set of customers similar to the customers receiving the new offer. Random assignment between the test and control conditions, which is relatively easy with lists of customers/online marketing, should ensure that the two groups are pretty similar before they receive the offers. For instance, websites can serve different pages randomly.
Make The Tests Smart
Tests have to be meaningful tests. This is critical to adopting a scientific approach. Unfortunately, “tests” are often run that aren’t really rigorous. Non-rigorous tests are especially worrying when they are described in scientific terminology. The terminology used, laboratories etc.., is suggestive of proper testing. None of the benefits of a proper test accrue if the setup isn’t rigorous.
To do as he suggests Davenport offers six +1 easy steps to running a test. In his article explains what happens at each stage.
- create or refine hypothesis
- design test
- execute test
- analyze test
- plan rollout
- rollout
The plus one step is important. You must create a Learning Library. After testing you should capture the insights for posterity. In other words, there is little point in gaining knowledge from tests if everyone just forgets about it as soon as it is finished running.
Thomas Davenport has quite a knack of explaining ideas clearly. Anyone planning to try testing at their firm would do a lot worse than look at his advice.
For more on testing see see here and here.
Read: Thomas H. Davenport (2009) Smart Business Experiments, Harvard Business Review, February.
