In presidential election polling 1936 stands out as a uniquely bad disaster. What caused the classic polling pisaster?
1936, The Worst Ever Polling Disaster?
In 1936 The Literary Digest made a prediction that Republican Alf Landon would beat the incumbent Democratic president Franklin Delano Roosevelt in a landslide. This rates as arguably the biggest disaster of opinion polling. And it being presidential election polling this was very high profile. Rather than a landslide for Landon instead Roosevelt earned a rather comfortable win. The Literary Digest didn’t just get the winner wrong, the poll was stunningly inaccurate. “…the Literary Digest poll gained an infamous place in the history of survey research” (Squire, 1988, 125).
Before starting it is worth noting this was not deliberate (as far we know). The pollsters were not trying to bamboozle with data (see here for more on lying with statistics). Unfortunately the pollsters just weren’t very good. (To be fair this was the early days of polling).
Sampling Was Highly Problematic
The conventional explanation is one of sampling. I have seen this explanation before and may even have mentioned it in a class. The Literary Digest sent ballots out to people it found on lists of telephone and automobile owners. In 1936 people with cars and phones were rich. (These were luxury products not the necessaries they are for many people today). Rich people skewed Republican. Rather than scientifically sample the Literary Digest had assumed that having large numbers of people responding would make up for the fact that they mostly sent the ballots to richer people. This was definitely a significant problem. A follow up survey showed that Landon supporters were more likely to have received a ballot than Roosevelt supporters. (Pollsters try harder nowadays, and generally do better, but no process is perfect).
Peverill Squire looked into the follow up survey. He noted that, while sampling was a problem, it wasn’t the whole problem. “..if all of those who were polled had responded, the magazine would have, at least, correctly predicted Roosevelt the winner” (Squire, 1988, 125). While, the people who received the poll were more pro-Landon than the general public they still had a slight Roosevelt preference. If all had responded the poll would have predicted the result of a narrow Roosevelt win.
Non-Response Bias Also Caused Issues
An additional problem came from non-response bias. There was a marked tendency for Landon supporters to return their ballots at a higher frequency than Roosevelt supporters. “…the 1936 Literary Digest poll failed to project the correct vote percentages or even the right winner not simply because of its initial sample, but also because of a low response rate combined with a nonresponse bias” (Squire, 1988, 131).
The 1936 Literary Digest poll had a bunch of problems. These problems went beyond the sampling issues that we always hear about. It is a salutary lesson on how we can get our understanding of the world wrong if we don’t know what we are doing. I’m not sure why Democrats seemed so shy about admitting to supporting Roosevelt in 1936 but their reticence made the Literary Digest look pretty silly.
Presidential Election Polling
Not saying it is perfect but modern presidential election polling is certainly better.
Read: Peverill Squire (1988) Why the 1936 Literary Digest Poll Failed, Public Opinion Quarterly, 52, 125-133