A second and final post on Alex Edmans’ May Contain Lies, looks at some advice he gives and highlights some of his useful stories. The key thing the general reader might want to take from the book is that claims such as ‘science shows’ are nonsense. Remember, research shows not to believe claims that science shows.
Research Shows Not To Believe Claims That Science Shows
There can be some pretty terrible headlines on the internet/social media. They grab your attention and claim to show that science has proved, for example, that red wine might be good (or bad) for you.
Phrases like ‘Research shows that…’, ‘A study finds that…’, or ‘There is clear academic evidence that…’ are commonly bandied around as proof, but they’re often meaningless.
Edmans, 20204, page 5
One of my favorite examples of this which Edmans notes is the idea that a university study shows something. It might be said, “a Harvard study…”. As he points out Harvard doesn’t release studies. Academics at Harvard do. There are some great academics at Harvard, but the entire faculty body hasn’t agreed on this study. Even at the best university, there are likely some pretty bad academics too. Maybe the study, that was said to come from Harvard, actually came from the embarrassing faculty member that other professors don’t like to mention. It is very possible that other professors at the university are waiting with bated breath till that particular tenure mistake colleague dies and they can get someone better in.
Interesting Bits Of Knowledge
Edman’s book contains some interesting bits of knowledge. For example, in 1863 Austin Flint conducted the first use of a placebo. This is where the person in the study receives something that should have no effect — i.e., not an actual treatment – to separate the impact of merely being in the study from the effect of the actual treatment being studied.
In the book he has an attack on Built to Last by Collins and Porras. This book used to be very popular in MBA curricula but the tide has turned. It is now hard to find an academic to defend it. It is almost a source of academic pride to disparage the book. Seeing this change I do wonder what I’m teaching now that I will be dismissing in a few years when we think about it a bit more.
Edmans recommends publishers feature a health warning for books that are disputed. I’m not sure I see the publishers’ incentives to trash a book that they are promoting but it’d be a fun thing to do. I can think of a few books that I’d put a big warning label on. For no reason at all, here is a picture of a book that Penguin sells.
May Not Contain Lies
Probably what I liked least about Edmans’ book was his title. I was left wondering if an editor had foisted it upon Edmans in the hope the book would sell more. Of course, research you see on the internet “may contain lies” but I doubt that is the major problem. More importantly, I doubt that Edmans thinks lies are the main problem in research. Instead, his advice often focuses on challenges like confirmation bias. This is where we believe things that we want to believe. There are certainly cynical authors sharing information that biased individuals lap up. Yet, it can also be biased individuals doing the research. They just did bad research, but they didn’t necessarily mean to lie to anyone.
I don’t agree with him that “‘Lie’ is simply the opposite of ‘truth'” (page 12). This statement started the book but honestly doesn’t really seem to fit with anything that comes next. The book talks about doing research badly, and disseminating bad research. Yet, doing research badly isn’t the same as lying even if you get it published. Why is this distinction important? In order to solve a problem we often need to be clear about its causes. People, rightly, get defensive when you accuse them of lying when they know that they aren’t lying. You are much more likely to persuade them to change their approach by noting the challenges they face and the hurdles that they have fallen at. Incompetence is not lying. Neither are desirable, but they are very different.
Should You Promote Your Findings?
Edmans is British and it can show at times. He suggests:
If you need to shout about the conclusiveness of your proof or the novelty of your results, maybe they’re not strong enough to speak for themselves.
Edmans, 2024, page 224
While I applaud the sentiment I have to say that I find the sentiment a tiny bit quaint. I’ve been in North America for more than a score of years now and I would say that you absolutely must shout about the novelty of your results. If you don’t shout then someone who has done a much worse study, perhaps using only their overactive imagination, will shout about theirs and be heard a lot more.
For more by Alex Edmans see here and here.
Read: Alex Edmans (2024) May Contain Lies: How stories, statistics and studies exploit our biases — and what we can do about it, University of California Press