What is the future of random probability sampling?

Random probability based surveys provide a strong theoretical foundation and underpin the basis of much of market research, but the industry has been experiencing a steady decline in response rates over the last few years.

Kirby Swales, director of research at NatCen, gave a presentation on a cross-industry strategic initiative between Ipsos, Kantar Public, GFK and NatCen to highlight the challenges of conducting random probability surveys in social research. The research analysed the response rate of eight large face-to-face cross-sectional government surveys.

Supported by the MRS, the Social Research Association (SRA) and the Campaign for Social Science, the initiative found that response rate has dropped an average nine percentage points in 15 years across long-running surveys such as the British Social Attitudes survey, BARB and the Crime Survey for England and Wales.

Swales highlighted an overall decline in trust in government and brands and survey fatigue as issues impacting response rate. Additionally, decreasing availability and accessibility as a result of people working longer hours, people becoming more reluctant to open the door to researchers, and the rise of gated communities were also attributed to the decrease in response rates.

While non-contact rates have remained consistent across BARB, the British Social Attitudes Survey and the Crime Survey for England and Wales, refusal rates have increased over the past few years.

More effort is being made by researchers to keep response rates up, leading to an increase in costs. Other challenges for fieldwork surveys include recruiting and retaining interviewers.

Swales said the initiative highlights the need for a ‘strategic response’ from the industry. “We think that we’re at a point in the industry that we need to have a strategic response and discussion with clients about how we maintain response rates and the cost and effort needed.”

With response rates going down and fieldwork costs going up as researchers try to tackle this, Patten Smith, head of research methods at Ipsos Mori, said attempting to increase response rates may not be the best use of survey resources in all cases. There may be better ways of addressing survey errors and spending the money differently, he said, and it is important for researchers to look at when minimising non-response bias is critical to conclusions, as often it isn’t.

“It’s not to say that efforts to maximise response rate can’t ever be justified,” he said. “There are occasions where they do tend to make quite a bit of difference, but [they are] particular sets of variables and where it matters to your conclusion.”

The researchers also presented two new approaches that combine traditional methodology with online surveys. Joel Williams, head of survey methods at Kantar Public, gave an overview of the company’s ‘address-based online surveying’ approach, or ABOS, which combines traditional survey methods with online, and has had better completion rates when compared to online-only survey methods.

NatCen has also developed a new approach, a random-probability survey that combines web and CATI. Swales explained the aim is to provide a high-quality alternative to non-probability surveys when robust data is important, but face-to-face fieldwork is too slow or expensive.

During a discussion at the end of the lecture, Swales highlighted the important role random probability based sampling methodology plays in providing an accurate trusted representation of social trends.

Referring to recent confusion over whether UK crime statistics are going up or down, he said: “I’ve got a lot of time for varied and creative approaches to data collection, but when you’re looking at whether crime has gone up or down, imagine what that would have been like without a random probability survey in the mix as well. It would have got 10 times worse.

“It is important to stand back and think about the broader issues of trust and goodwill in research and what happens when there is confusion when it comes to social trends.”

Methodology should therefore be taken seriously by those commissioning research, he added. “We’re trying to get across this message to funders and commissioners to take this really seriously, because we’re now at a stage where we’re going to change our data collection methods – we’ve got to, with digital technology – but that journey and how we manage it is really important.”