One of marketing academics’ favorite recommendations is to measure outcomes. Still, measurement problems make it hard for academics to take our own advice. Let’s give academics the benefit of the doubt and assume they genuinely want to measure their achievements. How then do we go about measuring research quality?
Measuring Research Quality
A major part of academic life is creating research. In an ideal world, this gets published in journals. (Often mine doesn’t but it is still great stuff). The problem is that there are a vast number of journals and they aren’t equally good. Or equally hard to publish in.
This means a simple metric like counting articles is a poor way to encourage excellent research.
It can be argued that such an approach measures productivity but not scholarship.
Mort and Colleagues 2004, page 52
It would presumably be easier for a talented researcher to produce numerous “easy” pieces than toil endlessly away on a great piece. Yet, great pieces probably tend to be more impactful than lots of little bits. (One great article is certainly easier to read than finding nuggets of wisdom strewn across many pieces). As such we need a way to rank articles.
Ranking Journals
One solution is to rank the journals articles are published in. This uses the assumption that journal quality predicts article quality. This seems mostly reasonable. Still, it just pushes back the problem to the challenge to ranking journals. This challenge is compounded by the fact that different methods are used in different journals. Some journals tend to feature advanced econometrics. Some feature lab experiments. While other journals specialize in theoretical pieces. How do you compare a journal that publishes theoretical math papers with one that publishes in-depth field studies?
Subjectivity is rife in assessments so researchers make a virtue of this by using the survey method. Ask people which journals are best. Mort, McColl-Kennedy, Kiel, and Soutar (2004) survey senior marketing academics in Australia and New Zealand breaking away from the North American focus of much of our knowledge about academic marketing.
Table 1 of Mort and Colleagues (2004) tells us that the Journal of Consumer Research gets a near-perfect 4.9 out of 5. The Journal of Marketing at 4.8 isn’t far off. Interestingly there was no one who couldn’t rate the Journal of Marketing. I.e. all the marketing academics had read it. Yet, 18% could not rate the Journal of Consumer Research. Which journal theory is better? The one that is slightly more respected? Or the one that is, presumably, more impactful given it is more widely read? I guess that is a subjective judgment.
For more on ranking and publications see here, here, and here
Read: Gillian Sullivan Mort, Janet R. McColl-Kennedy, Geoffrey Kiel and Geoffrey N. Soutar (2004) Perceptions of Marketing Journals by Senior Academics in Australia and New Zealand, Australasian Marketing Journal 12 (2) pages 51-61.