This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Find out more here

OPINION31 March 2016

Have opinion polls had their day?

Opinion Public Sector UK

On the day of the British Polling Council’s report on what went wrong with the 2015 general election polling, is it time to look at predictive markets as an alternative method?

Opinion polls are part of the fabric of modern politics, fuelling the symbiotic relationship between politics and the media. Some academics argue that polls don’t just reflect public opinion, but help to shape it – in some countries polls are regarded as so influential that they are banned toward the end of the electoral process.

It is only recently in the UK, following last year’s general election in particular, that people have begun to question the accuracy of such polls; for years they were seen as the best source of information available to us. But the opinion poll is being challenged by the emergence of a new academic tool: the predictive market.

Predictive markets present a real alternative to conventional research surveys.  They have been proven to be more accurate in predicting a range of outcomes and in addition to often being more cost effective, they avoid many of the issues that conventional research surveys are prone to, such as sampling bias, respondent fatigue and dubious responses to overly long or irrelevant questionnaires.

  •          Respondents self-select; they are free to answer whichever questions they want to, based on their own knowledge, from an array of questions covering different markets and subjects.
  •          They only respond to a question, place a bet, if they think they know better than the market – so no forced questions.
  •          There is a small amount of jeopardy.  If they bet incorrectly they will lose their stake – so more thought goes into each question.

A predictive market is the only true gamified research environment; respondents aren’t answering research questions, they really are playing. Analysis of the playing habits of participants in Predictive Markets has shown that they spend as long on the Predictive Market website as they do on social media sites and playing casual games such as Candy Crush and Angry Birds.

The chart below shows the betting on our predictive market as to whether the Scots would vote Yes in the Scottish referendum over a period of 10 months up to voting day. With the exception of the first four weeks, the outcome was never in doubt; the probability that the Scots would say Yes to independence was never higher than 50% and in the last three months was less than 30%. Even after the TV debates, the likelihood of a Yes vote only rose to 35% and that fell again to less than 30% shortly afterwards.



By contrast opinion polls showed an average of 43% Yes and 57% No (excluding don't knows) in July 2014 and August 2014 and in September, polls indicated that the vote would be even closer. On 6 September a YouGov poll gave those in favour 47% versus 45% for those against. But as the opinion polls narrowed and predicted a tighter outcome, the Predictive Market betting was moving decisively away from a Yes vote.

Similarly, the data below for the betting on the 2015 General Election show that, apart from some short-lived upturns, Ed Milliband never really got above 35% probability of being the next Prime Minister. David Cameron was seldom given a probability of less than 60-70%.


Probability of Ed Milliband being the next Prime Minister.

 


Probability of David Cameron being the next Prime Minister.

At the time of the general election the estimated voting outcomes published by most opinion polls were so close, 1% or 2% either way, that the differences weren’t even statistically significant. But, the predictive market was clear that there was only ever going to be one outcome.

By analysing many thousands of data points, Blinc and Media Predict (our US partner) have shown that predictive markets are far more accurate than traditional polling, and over a longer period, when it comes to election results. They have shown that women are no better at predicting outcomes for women than men are and vice versa! And they have shown that adults (regardless of whether they are parents) are capable of accurately predicting outcomes for children.

All of this is despite the absence of any adjustment to reflect the electorate or the national distribution of age, sex or anything else in the sample.

And predictive markets aren’t just useful for predicting political outcomes.  They can be used for optimising creative content, concept evaluation, NPD and a range of other applications.

In the US, the concept has been adopted by a number of leading companies to use in their business decision-making.  The last words on the subject go to the Harvard Business Review:

“Hewlett-Packard, Motorola, Intel, Best Buy, Microsoft, Google, and Pfizer have all employed internal prediction markets to assess likely product shipment dates, predict sales volume, and identify best-selling products. Their results have been impressive.” Harvard Business Review, January 2015

By Justin Charlton-Jones, managing director, and Paul Barrow, founder, Blinc

6 Comments

3 years ago

Very interesting. There is a concern that there was not enough discussion about the fundamental question - i.e. asking about intended behaviour in a situation where, more than ever before, voters were confronted by a range of largely unappealing options. In that context how much credence and reliability can we place on their responses? So anything that provides other mechanisms to predict likely outcomes is a welcome addition to the debate

Like Report

3 years ago

I'd been writing something quite similar myself on this topic! What struck me in thinking about it is that polls as they stand are an incredibly literal and complex way of assessing the likely outcome of the vote - like trying to estimate Coke sales by asking people how many bottles they buy. And as an approach, it's broken and needs to be fundamentally readdressed - sampling and methodology tweaks are no longer the answer.

Like Report

3 years ago

If someone could organise a predictive market that functions without any access to opinion poll data then we'd have a real test of how good they are

Like Report

3 years ago

Nick, you're missing the point. A predictive market is valuable because it responds to events and information in the public domain, which informs the decisions that are taken. If opinion polls are being published then they become part of the context that the 'crowd' uses to make its judgement; each individual decides whether or not the information is useful to them and they will not all have the same information. The 'crowd' who responded to the questions we asked on the general election clearly felt that they knew better than the opinion polls and they were proved to be right.

Like Report

3 years ago

YouGov predicted a 46% YES vote in the Scottish Referendum - the final count showed a 45% vote, which seems pretty close to me. In contrast, Predictive Markets seemed to suggest a final 30% YES vote, which is more than double the error rate witnessed in the national election. On that evidence, I'd need some more convincing to adopt this methodology, although I'll still fling the term about in meetings, obviously.

Like Report

3 years ago

Russell, the participants were asked about the likelihood of a vote for Scottish Independence, not what the percentages of the actual poll would be, although we could ask that question. The predictive market data showed that for 10 months prior to the vote the expected outcome was that the Scots would not vote for independence and that outcome became ever more certain as the polling date approached. The 30% figure is the number of people betting on a 'Yes' vote. Given the relative closeness of the referendum, the certainty of the outcome was the real benefit of this approach.

Like Report