Are opinion polls fit for purpose?

That was the question faced by pollsters and statistics experts on the first day of the MRS’s annual conference, Impact 2017, in London yesterday.

Both the Brexit and Trump results seemed to reveal how the media, political elites and pollsters had failed to understand how voters were really feeling about some of the biggest issues of the day. Jil Matheson, who formerly served as national statistician, was left wondering whether "any of us really understand the public mood".

Martin Boon of ICM said his first response when asked why polls have failed is always, “Why are you asking me? I’m only a pollster.” Boon agreed that the 2015 election polls were “a disaster”, but said the Brexit vote was simply too close to call (and that history has forgotten how many polls did in fact point to a leave victory).

However, Boon said pollsters who tweaked their methodologies in their final Brexit polls were “culpable”. “There were eight companies which produced a poll, and six of the eight changed their methodology for their final polls,” he said. “Guess how many of those six changed them with the effect of increasing the remain margin of victory? All six. What we pollsters succumbed to there was this narrative of fear – this Westminster-centric elite had decided for whatever reason that remain were going to be the winner, and it’s the natural instinct of a human being to respond in a way that makes you think you’re going to improve your outcomes in line with your prejudices. I think what happened there was that panic buttons were pressed, and new techniques were employed that made it universally worse than it would have been.”

Sometimes, Boon said, the decisions that pollsters must make about which methodology will best predict the outcome, come down to “gut instinct”.

Matheson had limited sympathy for this view, saying: “If I had sat in the ONS and said, it’s my gut instinct and I’m going to adjust the methodology, then I would have had very short shrift and quite rightly. So there is something here about you’ve got to be transparent and consistent in your method.”

We are transparent, the pollsters insisted, even if, in the case of the Brexit vote, they weren’t consistent.

Nick Moon of Moonlight Research defended the performance of polls in last year’s US presidential election, saying: “Let’s not forget the polls said Hillary Clinton was going to win [the popular vote] by 3%. She won by 2%. The polls were incredibly accurate throughout in terms of the popular vote, and most of the state polls were accurate."

The problem was that "the states that were inaccurate were all the states that mattered".

Moon also said that the “feedback loop” created by media coverage of polls during campaigns can be "dangerous". For instance, if the 2015 election polls hadn’t pointed to a hung parliament, the media coverage might have been less dominated by speculation about a deal between Labour and the SNP, and the Conservatives might have faced more scrutiny. As for last year’s US election, “you can make a fairly strong case that the polls cost Clinton the election, because if they had been in any way accurate in those key swing states, she’d have been spending resources there,” Moon said.

In response to a question from Corrine Moy of GfK on whether polls presented an impossible task, Boon said it would be “silly” to give up on them completely. “I absolutely disagree that it’s more trouble than it’s worth [conducting opinion polls]," he said. "But I would say that, wouldn’t I.”

Jil Matheson agreed that polls have their place, saying: “What else have we got? We’ve got the prejudice of newspaper editors, or the assertions made with great confidence by politicians of what’s going to happen, and no basis to challenge them.”