Meta Analysis: Combining Many Results into One

How marketing impacts firm value is an important question facing for-profit businesses. In an ambititous paper Conchar, Crask and Zinkhan (2005) examined this. The hard work involved in this paper is impressive. The authors looked at 88 studies of advertising and promotion and noted the relationship to what they say are measures of market value. They conducted a meta-analysis. This is a way of combining many studies into a single study, a technique which seems especially popular in psychology. The basic idea is that we can see whether any single study was an anomaly by looking at several similar studies and comparing the outcomes.

Unfortunately, the way academia is set up mitigates against the success of many meta-analysis. I don’t want to be too negative, often meta-analyses can add new insight, but I worry that Conchar and colleagues suffered from the typical problems of meta-analyses while adding some new ones. Given they can only analyze studies that have already been done they are limited by the quantity and quality of past work but that is true of all meta-analyses.

Further common challenges include the file drawer problem. Generally only successful tests get published. Journals, for understandable reasons, don’t want to print that Professor XYZ did a load of work which completely failed to get an interesting result. If you just use reported data, therefore, it looks like everything worked. Best practice, and to be fair the authors did this, is to ask for unpublished studies. This is a sensible approach but even the hardest working researcher who reaches out to everyone possible would be unwise to suggest that negative and positive results have an equal chance of representation. The dramatic or novel is always more likely to be reported.

I think another problem of academia is probably most challenging for these particular authors. It is very hard to win plaudits for “replications” — doing the same thing as someone else. We all want to do exciting new stuff. This means each study has to be a little different to be published. In the Conchar piece this means their market value measures are nearly all different. Given the outcome each study predicts is different I’m not sure what it means to put them together into the one analysis.

To see the problem think of market value as analogous to the size of person. We want to see what is associated with size. Unfortunately the construct of size is measured in many different ways, e.g., height, weight, comparison to the average person in your area, comparison to average person of the same gender worldwide, etc… The inputs are all different too, we look at men, women, kids, babies, older people etc… The resulting connections will be extremely messy. Babies might get taller with more food but adults won’t, where you live might be associated with size but not if you correct by local averages. To make sense of it we need some sort of theory of what is expected to happen and that is largely missing. The authors largely admit this: “the choice of the most appropriate criterion measure should ultimately be grounded in theory rather than based upon measurement considerations” (Conchar, Crask and Zinkhan, 2005, page 456). In academic speak this means don’t just throw a bunch of stuff you don’t understand into a statistical model and see what happens.

In their model they can’t speak to causes. So if they see high marketing spend associated with high market value they aren’t sure if marketing increases market value or having a high market value allows you to splurge on marketing. This seems like it is a pretty crucial distinction but I don’t want to be too negative, I admire the work put in.

Read: Margy P. Conchar, Melvin R. Crask, and George M. Zinkhan. (2005) “Market valuation models of the effect of advertising and promotional spending: a review and meta-analysis.” Journal of the Academy of Marketing Science 33 (4) pages 445-460.