Reliability in analysis is widely valued. Reliability being returning the same result whenever we test the same thing. I’m not against reliability but it can be overrated as J. Scott Armstrong explains.
The reliability of a measurement tells us whether we are being precise in what we are doing. When we are testing something, and looking for reliability, we seek out “…approaches that are as similar as possible.” (Armstrong, 1974). This is a typical academic approach, we seek to get a very precise read on something that is often quite small. Academic studies using surveys tend to ask almost exactly the same thing several times and then celebrate that people answer the questions similarly.
While it is true that getting random responses to a survey means you have written a terrible survey, the opposite isn’t true. Just because you get reliable responses doesn’t mean you have a good survey. You could, after all, be reliably measuring something that is: A) completely different to what you think you are measuring, or B) completely unimportant. A perfectly reliable measure that is misunderstood or trivial doesn’t do much to improve knowledge in the world.
Armstrong suggests that the solution is more ecletic research. Such research uses lots of different methods to triangulate upon a valid measure. This approach doesn’t look for a perfect measure but a number of good measures in general area of something important. When assessed together the measures give you confidence you are getting near the important thing you want to study.
To perform such research the academic needs a wide skill set, to be a “generalist”. Unfortunately all the incentives for academics are to be specialists. Armstrong suggests that science will progress faster if we focus at least as much on the validity of our measures as their reliability. It is a shame that 41 years after Armstong wrote I’d say the incentives for academics haven’t changed, we still prize reliability over validity
Read:J. Scott Armstrong (1974) Eclectic Research and Construct Validity, published in Models of Buyer Behavior: Conceptual, Quantative, and Empirical, (pages 3-14), edited by Jagdish N. Sheth, Harper collins, New York, NY