One of the most ‘interesting’ academic papers of recent times is Daryl Bem’s paper in the Journal of Personality and Social Psychology (a good journal) on psi. (Yes, psychic phenomenon). He purported to show that people had the ability to ‘feel’ the future and so predict it better than chance. This violated our standard notion of time going forward.
This paper set a major challenge for experimental research. Bem, a well respected figure, demonstrated support for something that most people think is nonsense. Is the problem with our closed-minded views? Is the problem with lab experiments? Can you “prove” anything you want to with a lab experiment?
There are a number of potential hits against Bem’s work from a methodological perspective. I would, however, say that Bem’s paper is certainly not obviously worse on this front than many other papers. Clearly he got put under more of a microscope but I’m not aware that he did anything worse than many others methodologically. There are certainly tricks experimenters use that cast doubt on a lot of papers but Bem didn’t invent them. Since then experimenters have moved (a little, it could be more) towards better practice so in many ways the paper did everyone a great favour.
That said the methodological arguments seem to be secondary to me. Being honest I don’t think he could have done anything in the lab to substantially move my disbelief in psi. Rather than call myself closed minded I’d prefer to refer to my strong priors. I had a very strong prior that psi doesn’t exist and maybe Bem moved me a tiny, tiny, tiny, tiny, tiny bit towards thinking maybe there is something there but the move was basically imperceptible. I still don’t believe in psi in any measurable way whatsoever.
I would say that the lack of theory is a fundamental problem with this (and a lot of other work). Without a proper theory of how the process might work any empirical findings are not very compelling. Without a strong theory of how we might feel the future experimenter error seems a lot more plausible than psi as an explanation for the findings.
I’d take comfort by noting that Bem suggests that many would agree that “‘extraordinary claims require extraordinary evidence'” (Bem, 2011, page 407). The problem with Bem’s paper is that it is very ordinary. I don’t mean that as an insult, just an observation. It is ordinary in the sense that it does what people in the field usually did. It is an exclusively lab based demonstration. It doesn’t draw in work from outside its field except in a rather superficial way and doesn’t support the lab work with any other type of evidence.
Psychologists were right to work on improving their methods post-Bem’s paper but I think we all benefit from getting out of our methodological silos. There is life (but probably not psi) beyond the lab.
Read: Daryl J. Bem (2011) “Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect.” Journal of personality and social psychology 100.3 407-425