Here’s to the next ten years

What do the next ten years hold for government social surveys? Will a pernicious mix of public expenditure cuts, hostile ministers, the rise of administrative data and social media and privacy fears herald their demise?

I doubt it. But things will change. Here are three predictions:

1. The government-commissioned social survey will thrive, despite pressure.
The first year of the coalition government was pretty tough for everyone in the market for government social surveys. Government social researchers slammed on the commissioning brakes, cancelled contracts and generally became distracted by restructuring. But the dust is settling and things are becoming clearer.

Budgets are smaller; there have been some high-profile survey casualties; some ministers delight in criticising survey research. But equally, the positive signs are strong. The science minister has approved the 2012 birth cohort study. Major surveys are being re-commissioned, including the English Housing Survey and the British Crime Survey. Why? Because even where there is no shortage of administrative data (for example police crime statistics) the resulting picture is limited.

Administrative data has an absolutely critical role and can be used more effectively. But social surveys are still proving necessary for consistency, completeness and ensuring all sections of the population are properly represented, together with their capacity to connect different aspects of people’s lives. What’s more, even ‘small state’ governments need to know where the problems lie in society and whether their policies are working.

So I predict that the future of the social survey will be a bit like radio. Twenty years ago people assumed radio’s eventual death because of the rise of alternative media technology. In fact radio has flourished. It has used the new technologies to add new distribution channels and to interact more imaginatively with listeners. The social survey will be similarly enhanced through new modes and by linkages to administrative and other data.

2. The market will open up and fewer government commissioned surveys will be delivered by the public sector.
It’s no real surprise that government continues to need the social survey. But one oddity is how much is still delivered by a publicly funded body with no requirement to compete for that work. I have huge respect and affection for Office for National Statistics’ (ONS) Social Survey division. I trained there and know just how good its people are. ONS has driven forward an admirable modernisation agenda including the move to Newport, more use of technology and experimentation with social media.

But no one nowadays expects our publicly funded physical infrastructure to be built by nationalised building companies. So why is so much of our data infrastructure delivered by a public body? I predict that the incoming chair of the UK Statistics Authority will look hard at ONS’s commercial strategy and review what should continue to be conducted in-house and what should be outsourced or put out to tender. At the same time, I expect that ONS will play a more starring role across government as a trusted player, upholding standards, and facilitating much needed coordination.

3. Participants will expect more in return for their co-operation and we’ll have to respond.
Maintaining decent response rates remains a challenge; it was ever thus. What’s perhaps more surprising is the astonishing goodwill shown by participants. Many take part in social surveys for a relatively modest financial incentive because they believe it will do some good for wider society. And interviewers know this: they tell us that being able to provide compelling examples of how the data is used is the best motivator. But don’t we rather stretch this goodwill with often long and arduous questioning? I think that over the next decade, we’ll realise (not before time) that we need to do more for those taking part. I predict that we’ll see more consultation with the public on the objectives and content of the major research projects. And I believe that we’ll enrich the interview experience and get much better at providing information to participants about the results the use of their valuable time generates.

Penny Young is the chief executive of NatCen, the National Centre for Social Research

• This article is the last in a four-part series. Click for parts one, two and three