NEWS5 June 2013

Gallup review IDs flaws in 2012 presidential polls

Features North America

US — Gallup’s review of last year’s presidential election polling has identified four factors that “most likely contributed” to the difference between the company’s estimate and the final vote outcome.

The research firm’s polls hinted at a win for Republican challenger Mitt Romney in the hours before Barack Obama won a second term in office.

In a report published yesterday, based on work by Gallup’s editor-in-chief Frank Newport and others, including the University of Michigan’s Michael Traugott, the company pointed the blame at its estimates of likely voters, its regional controls on interviews, the way it asked about race and ethnicity, and its use of a listed landline sample frame instead of an RDD list-assisted landline sample frame.

On likely voter estimates, Gallup said that while its procedures are “broadly similar to those of other survey organisations”, its likely voter questions are “more heavily weighted towards past voting behaviour”. But the more significant variable was the ‘thought given to the election’ variable. Removing this, along with other changes, would have made Gallup’s final estimate more similar to other firms, the company said.

Regarding its regional controls, Gallup said interviewing factors “can influence the percentage of completed interviews obtained in geographic units within each region”. “As a result,” it said, “completed interviews by time zone within a region can be disproportionate, even if a region is appropriately represented.” In the case of the pre-election surveys, this meant that on occasions the Eastern time zone within the Midwestern and Southern regions, and the Pacific time zone within the Western region were underrepresented.

Gallup said it has already changed the way it collects data about race and ethnicity. During the election campaign, it used a series of forced-choice yes/no questions for each of several race and ethnicity categories. This resulted in “a disproportionate number of respondents reporting they were multiracial and American Indian/Alaska Native”. Gallup now reads respondents a list of all racial and ethnic categories at once, and respondents can select up to five categories.

On the sample frame used, Gallup said: “Experimental research showed in the spring of 2013 that on an unweighted basis, the listed landline sample in use in 2012 consisted of older and more Republican respondents than the RDD list-assisted landline sample.” In addition, it said: “The listed landline and cellphone sample frames also underrepresented dual cellphone and landline users whose landline was unlisted, which required significantly more weighting of respondents interviewed on cellphones.”

The full report is available online here.

Gallup was polling for USA Today during the election but their 20-year partnership ended soon after. Commenting on yesterday’s report, USA Today’s editor-in-chief David Callaway said: “USA Today had a long and successful relationship with Gallup. Any time we hear there may be problems with information we rely on, we are concerned. But we trust Gallup, like us, conducted its polls with the utmost integrity.”

USA Today is now working with the Pew Research Center.