Reliability in analysis is widely valued. Reliability is returning the same result whenever we test the same thing. I’m not against reliability. Still, it can be overrated as Scott Armstrong explains. We need to ask occasionally: Is Reliability A Good Thing? Or more accurately, how much are we willing to sacrifice at the altar of reliability?
Looking For A Good Measure
The reliability of a measurement tells us whether we are being precise in what we are doing. When we are testing something, and looking for reliability, we seek out “…approaches that are as similar as possible.” (Armstrong, 1974). This is a typical academic approach. To get a very precise understanding of something that is often quite small. Academic studies using surveys tend to ask almost exactly the same thing several times. The authors then celebrate that people answer the similar questions similarly.
Is Reliability A Good Thing?
It is true that getting random responses to a survey means you have written a terrible survey. That said, the opposite isn’t true. Just because you get reliable responses doesn’t mean you have a good survey. You could, after all, be reliably measuring something that is:
- completely different to what you think you are measuring, or
- completely unimportant.
A perfectly reliable measure that is misunderstood or trivial doesn’t do much to improve knowledge in the world.
More Eclectic Research
Armstrong suggests that the solution is more eclectic research. Such research uses lots of different methods to triangulate upon a valid measure. This approach doesn’t look for a perfect measure. Instead, it looks for a number of good measures in the general area of something important. When assessed together the measures give you confidence you are getting near the important thing that you want to study.
To perform such research the academic needs a wide skill set, to be a “generalist”. Unfortunately, all the incentives for academics are to be specialists. Armstrong suggests that science will progress faster if we focus at least as much on the validity of our measures as their reliability. It is a shame that 41 years after Armstong wrote I’d say the incentives for academics haven’t changed. We still prize reliability over validity.
For more on academia see here and here.
Read:J. Scott Armstrong (1974) Eclectic Research and Construct Validity, published in Models of Buyer Behavior: Conceptual, Quantitative, and Empirical, (pages 3-14), edited by Jagdish N. Sheth, Harper collins, New York, NY