‘Unrepresentative’ samples led to Australian polling ‘failure', finds inquiry

The inquiry into the performance of opinion polls prior to last year’s election was led by the Association of Market and Social Research Organisations (AMSRO).

The final report, published today, found that the polls erred in their estimate of the vote in a way that was ‘statistically significant’, and that they erred in the same direction and by a similar margin.

All of the national election polls published during last year’s campaign showed that the Labor party had the support of the majority of Australian voters (in terms of the two-party-preferred vote), however, the election was won by the Coalition, with 51.5% of the vote, compared with Labor’s 48.5%.

The source of the errors lay in the polls themselves and was not due to a late swing in voter preference, the inquiry found. It also ruled out the impact of so-called ‘shy conservatives’ and voters misleading pollsters as contributing factors.

“Our conclusion is that the most likely reason why the polls underestimated the first preference vote for the LNP and overestimated it for Labor was because the samples were unrepresentative and inadequately adjusted,” the report said.

The polls were likely to have been biased towards the “more politically engaged and better educated voters”, according to the inquiry, and not adjusted for this, over-representing Labor voters as a result. 

Australian pollsters had, in aggregate, a 96% success rate across the four previous federal elections between 2007 and 2016, the report noted.

The report said that while the methodologies used by Australian polling companies differed, they shared a “common difficulty in struggling to establish contact with and gain the cooperation of a representative sample of voters”, and added: “This conclusion is broadly similar to that reached by the reviews into the performance of the 2015 UK polls and the 2016 US polls.”

In the US, the polling industry is again under scrutiny after a number of polls forecast that Joe Biden would win the presidential election by a bigger margin than he did.

The Australian inquiry has recommended that a code of conduct for election polling is established to provide oversight, regulation and disclosure. It also recommended an approach similar to that of the British Polling Council to ensure compliance.

Pollsters should develop “more effective” sample balancing and/or weighting to improve representativeness, trial new calculation methods for the two-party preferred vote, and clarify measures of uncertainty, the report said.

The inquiry also found some evidence that the reporting of polls failed to consistently meet disclosure guidelines set out for journalists by the Australian Press Council. An interim report published by the inquiry earlier in the year proposed 24 disclosure standards.