Survey Methodology and the Future of West Ham United

West Ham United, the English football (soccer) club I support, hasn’t seen much recent success. The fans, in a fine example of optimism bias, expect the team to win while playing with a certain élan. In 2011, after a disastrous few years, Sam Allardyce was installed as manager. Results haven’t been magical but under Allardyce West Ham have been promoted to, and survived two seasons in, the top tier of English football. The supporters aren’t universally happy with the pragmatic style of play adopted. A number of fan websites therefore collaborated to ask about Allardyce’s future. I use this survey to illustrate problems with methodology.

Firstly, choice of question has a surprisingly strong impact on the result. The question used was: “Should Sam Allardyce be retained as manager of West Ham for the 2014-15 season?” I’d suggest that this is not the right question given Sam Allardyce has a contract to manage West Ham for the year. The choice isn’t really whether to “retain” Allardyce but whether to fire him as manager, i.e. break his employment contract. A survey’s wording should reflect the reality of what is suggested.

Many will have little sympathy for Allardyce if he is fired because, although managers get fired at an alarming rate, they are usually given massive payoffs. Even if you don’t have qualms about breaking the employment contract firing a manager costs several million pounds. A better question, therefore, might be: “Would you like to fire Sam Allardyce instead of recruiting a multi-million pound player?” Similarly, no replacement manager is mentioned. The decision the board are faced with making is between Allardyce and any alternative managers who would take the vacant role. The fans consultation question doesn’t mirror the board’s decision so is of limited value in informing the decision.

Secondly, the survey was conducted after a string of disappointing results on the pitch. Survey timing matters. When considering a survey always ask: Did people think the same last month? Will they feel the same next month?

Thirdly, the respondents were obtained from 16 supporter websites. Many of the websites are covered with stories criticizing Allardyce and his style of play. Polling stations are supposed to keep potentially biasing materials away from voters, websites do the exact opposite.

Finally, recruiting volunteer respondents through websites does not give a random sample. The website’s visitors are probably atypical of all supporters and those filling in the survey may not even be typical of those who visit each website. Non-scientifically selected samples are dangerous — they appear useful but the results are always highly suspect.

In the poll 77% voted against Sam Allardyce who maintained that only a minority of supporters wanted him fired. His critics were infuriated but Allardyce was right to say that the evidence against him was weak. We really can’t conclude much from this survey. Allardyce, even if fired, will always be welcome in my class as a guest speaker critiquing survey methodology.

Read: One of many West Ham fan sites. My favourite is West Ham Till I Die: