OPINION17 December 2013
OPINION17 December 2013
Asking strangers about their sex lives isn’t easy – especially when you’re wanting truthful answers. Penny Young explains how NatCen’s interviewers broached the subject for the third Natsal survey.
Ever asked a stranger whether they have paid for sex, or masturbated in the last four weeks? No, me neither.
Yet, over the past few years, NatCen Social Research survey interviewers have been doing just that. And they’ve managed to get more than 15,000 people to answer.
The third National Survey of Sexual Attitudes and Lifestyles (Natsal) was published in November, and you will probably have seen some of the headlines: people in the UK are, on average, having less sex than in the past; women have more sexual partners, and many people are having sex well into their 70s.
A lot of our work at NatCen involves eliciting sensitive information from people in a wide range of contexts, but this study is a pretty extraordinary one. It throws into sharp relief two enduring survey research problems: how can we be sure that our sample is representative of the wider population and how sure are we that people are telling the truth?
The survey achieved a response rate of 58%, a similar level to many other less contentious surveys – and this is a great achievement, given the sensitive subject matter. Increasingly, we worry about response bias and not just the overall response rate, so (after weighting for age, sex, and geographic region) it was reassuring that our sample was a good match to the 2011 census figures, in terms of marital status, ethnicity, and self-reported health status.
“Tangible incentives are often important to encourage people to participate in social surveys, but if you can’t be convincing about why it’s important to take part then you won’t get a decent response rate”
Part of achieving these kinds of response rates is about doing the simple things well, like using professional looking materials. Interviewers are also essential in persuading people to take part, and they need to believe in the purpose of the survey in order to sell it.
The academics on the study have been instrumental here – Professors Kaye Wellings and Anne Johnson have always been clear that the study is a collaboration between academics, researchers and interviewers. What’s more, they walk the talk: they have been involved in briefing interviewers, and explaining how previous waves have brought about real changes in sexual health policy, and how the study is highly respected by other researchers and practitioners.
Increasingly, our interviewers are tough with us about what the point of any survey is – and we believe that while tangible incentives are often important to encourage people to participate in social surveys, if you can’t be convincing about why it’s important to take part then you won’t get a decent response rate.
One of the questions that people always want to know about a study like Natsal is, ‘How do you know that people are telling the truth?’
As with any other survey, we are reliant on peoples self-reports. When talking about sex, it is generally assumed that people will be less willing to answer honestly. Equally, people may feel embarrassed about not having sex. However, there are many ways that we create an interview environment that encourages people to answer accurately, using methods developed over the course of the three surveys.
Question wording is a particular challenge in sex research, as sex is still a taboo subject which people are not always comfortable with. Before the first Natsal survey in 1990, researchers at NatCen (then SCPR) carried out qualitative interviews to find out what kind of language people wanted researchers to use when asking them about sex.
Since then the Natsal questionnaire has evolved to keep up with this rapidly changing area, and to meet the demands of the various users of the data. For each new survey the questionnaire is scrutinised and pre-tested, to ensure that it stays acceptable and understandable.
It goes without saying that all participants are reassured that their answers will be treated confidentially. Key to maintaining privacy and enabling people to answer accurately is the fact that we use computer-assisted self-interview (CASI) for the more sensitive topics (around half of the interview questions). The interviewers also try to make sure that the interview is carried out with other members of the household out of earshot. This was achieved in 71% of interviews for Natsal-3.
The interviewers set the tone for the participants, so it was really crucial for them to be approachable and matter of fact. Although we minimise the role of the interviewer by using CASI for the more sensitive questions, it’s clear that our interviewers play a key role in eliciting accurate data.
In methodological work, where we followed up with participants from the second Natsal survey, people told us that one of the reasons they had reported their behaviours honestly was because it was important to be truthful in a scientific study and they trusted the research organisations involved. Much of this is communicated to participants by our interviewers.
So, do we feel that our results are accurate, and that people have answered honestly? It’s difficult for us to ever be 100% sure. However, there are several things that give us reason to believe that people generally answer honestly.
Firstly, using the methods above we know that we have done everything possible to create the right environment for accurate reporting.
Secondly, we are able to check our data both for inconsistencies and missing data. We ran over 130 consistency checks on the Natsal-3 data and found that most interviews were internally consistent. We also found low levels of missing data (people refusing to answer or saying ‘don’t know’): typically less than 3%, even for the most sensitive questions.
This is much lower than the level of missing data for the one really problematic question in the survey – the one that asked for participants’ income, where around one fifth of participants didn’t answer. So it’s money, not sex, that the British public feels most uncomfortable talking about.
Penny Young is chief executive of NatCen Social Research