In development communities, SMS research is currently used in a number of scenarios. The cost-effective approach – useful because it works on any phone and has the potential to reach formerly inaccessible populations – involves individual text (SMS) messages being sent to respondents’ mobile phones. If that person responds, they receive another question and, if they respond to that, they receive another, and so on.

It has been used with relative success to research existing defined populations – such as school headteachers and health workers reporting on school quality and medicine stocks, for example – for rapid evaluation, or for gaining insight from people on the ground during times of crisis.

But when it comes to population-based surveys – that is, surveys representative of an entire population – the approach is not well tested. Melissa Baker, regional director for public affairs research for Africa and the Middle East, alongside Dr Charles Lau, survey methodologist at RTI and Dr Ansie Lombaard – global technology enabled research (TER) innovation director at TNS, set out to test whether it was a viable way to collect information that would be representative of an entire population.

The team carried out research across Kenya, Ghana, Nigeria and Uganda with two key objectives. The first was to ascertain sample representation validity– that is, comparing how close a mobile phone-only sample would be to a sample including mobile phone owners (achieved using a face to face sample). The other objective was to test a number of design features of SMS surveys to find ways to improve the response rate to questionnaires.

On the first objective, the findings were fairly clear: representativeness is not easily achievable via the SMS method. “Because of very low response rates from SMS population surveys like this, where we’re basically sending the SMS to people that aren’t expecting it, that introduces quite a bias in the sample,” says Baker. “In development work, we want to make sure we’re covering a representative sample, especially including the poor. And SMS really doesn’t do that – it has a strong bias towards those with a secondary education or higher.”

For example in Kenya, compared to the face-to-face sample, SMS respondents were more likely to have more than one SIM card, less likely to have anyone else use their SIM card, and more likely to be aware of and use the internet.

There was another key difference, says Baker. “For some of the countries it’s quite legal just to send out an SMS to anyone with a phone,” she says. “But, for other countries there’s a kind of opt-in process.” In Kenya and Uganda, where network providers had previously gathered demographic data from people and had some form of opt-in process, response rates were much higher than in Ghana and Nigeria, where messages were sent out on a blanket basis.

Looking more generally at the research design, it was found that incentive levels and questionnaire length had very little effect on response rates. The team tested two different incentive levels, and found that the response rate was hardly different if respondents were paid 50c or  US$1.25 – though Baker admits that stepping the rate up significantly could increase response rates, but this wasn’t tested.

Looking at questionnaire length, although the majority of people dropped out after the first couple of questions, if they stuck with it beyond that then the team achieved the same response rate for 16 questions as for eight questions.

The big lesson from the study, says Baker, is that if you want a representative sample you can’t use SMS. But, she says, things are changing fast.

“We’ll be keeping an eye on mobile uptake – people are gradually getting more familiar with SMS in sub-Saharan Africa. But in countries where we have crises going on and we need to get some kind of response – for example getting into Somalia or other war zones – SMS is obviously really useful for that. It’s maybe the only way of getting information quickly from a large sample on the ground.”