FEATURE29 May 2013

A second chance to get it right

Mistakes were made in the move from telephone to online research. Now, as work transitions from online to smartphones, Melanie Courtright warns against making the same mistakes twice.


As the research industry gets swept-up in everything mobile, researchers typically apply the traditions and lessons learned from mainstream practices to new methodologies. But does it work to apply the same rules? As the mobile business at Research Now has developed and grown, we’ve carried out several “research on research” studies in an attempt to answer these questions with science and measurement, rather than relying on assumption and tradition.

Overall, our learning from this has been that mobile is indeed different and studies must be designed with this in mind, in order to get the best response. Does this sound familiar? A new survey mode with higher respondent engagement due to ease and experience, but which comes with the need to adjust our instruments and better our practices. These findings mirror what we noticed in the migration from phone to online.

During that time, we had the opportunity to reinvent the surveys we presented and the ways we went about collecting data. Many believe we squandered that opportunity; placing research concerns about comparability ahead of the survey takers’ need for a better experience, and sacrificing one error type for another. Here’s my take on what we can learn from the move from phone to online, and how to apply it to mobile so as not to make the same mistakes twice.

1. A change of medium means a change of survey design

When we moved from phone to online, we did not initially understand online as a different medium, and we did not therefore change the way we asked people survey questions. We took the same questionnaire methods we had been using from paper and phone, and just altered them to be self-administered. This created a disconnection between an old survey style and the new web environment. With the switch to mobile, this is a vital issue. It’s not just about a tiny screen driving editorial brevity, but also about how to engage with customers on mobile and design an interface which is easy and enjoyable to work with.

The assumption would naturally be that you can’t ask mobile participants to do very much – mobile surveys have to be very short and very easy. People interact with their mobile devices, but in smaller increments of time than on a computer. To test this, we reviewed the more than 150 mobile projects we’ve completed at Research Now, and compared the completion and response rates to each other and to what we would typically see from an online survey.

What we found was that response rates are very healthy for mobile surveys, indexing higher than online survey response rates. We put this down to a novelty lift, as we saw in the early days of online. But, in terms of content, we found that surveys asking respondents to do too much (multiple audio and video tasks, for example) or engage with content that is uninteresting or hard to remember, results in lower completion rates. The survey format has an impact on completion rates as well. Trying to retain online questions in a mobile environment, resulting in difficulty manoeuvring through the survey, leads to lower completion rates. Finally, surveys that take longer than 15 minutes, often due to the survey content and format issues we have just described, also have lower completion rates.

2. Reaching out

Of course the move from phone to online revolutionised the whole market research industry in terms of sample and panels; many in the industry were slow to understand the full impact that it would have. The move from online to mobile can be even more radical due to the various types of information that can be gathered from a mobile respondent.

“We have been presented with another opportunity to reinvent surveys and how we go about collecting market research data. Let’s get it right this time around”

People see their phones as a personal extension of themselves. Downloading an app to store on your smartphone requires more commitment from a punter than a passing pop-up or even signing up to a panel. It’s like giving a little bit of your private phone space over to the research company in question.

The personal nature of mobile means that the way we get in touch with participants to invite them needs to be very carefully thought out. The communication we use, the introductions we script, the reliance on one form of invitations, the standard response rate measures; all have to change.

3. What we expect of people

When we went online, there was a change in what we could expect of survey participants. We suddenly realised that without the human touch of an interviewer, participants got bored and lost track more easily, and long questionnaires were much harder to field. Without a person to remind them, complex questionnaires which required participants to remember what they answered to previous questions or which did not easily direct them according to their responses were impossible. Faced with large grids and extensive scales, the new online respondent switched off. The same applies to mobile, but again, it’s not just about being short and sweet. It’s also about being responsive to the device on which they are working.  

Mobile allows us to touch in with participants; to ask questions in bursts at specific moments and in specific situations. This gives us enormous richness, that ploughing through a lengthy online survey on a PC simply cannot deliver. This new device for collection also allows us to get much closer to the event we are researching – the experience they had in the store they just visited; the events taking place during their shopping experience; what they remember about the advertisers in the show they are watching.

We should also think about what we expect people to answer about themselves over time. Perhaps the time is right to find a way to avoid asking age and gender every time they take a survey. Or use behavioural data to understand their recent purchases and store visits, rather than asking them to remember where they shopped. Or review our members’ digital downloads, site visits and geo-locations – all with their permission, of course – to determine who is in the market for a new car. Reducing the cognitive burden on respondents should be a key focus for us as we evolve our surveys.

4. How we thank people

We’ve known for decades that there are a number of reasons people participate in research. To give something back. To get something back. To pat themselves on the back.

Those who want to give back feel a sense of responsibility to help the world be a better place. Those who want to get something back are motivated by a return on their investment of time and energy. And those who want to pat themselves on the back sense that they are an expert in certain topics and enjoy sharing that expertise. It is important that we have all types of people and motivations represented in research. So when we recruit members and respondents, we use all types of messaging.

In mobile research, we are presented with new opportunities for thanking our participants. Mobile apps make it possible to more readily share content, games and data. The technology also affords more convenient ways to manage preferences and settings on their accounts. Mobile apps may allow us to become more ingrained in our members’ lifestyles and habits, helping us to secure the limited member resources and improve their experiences. The flexibility of the rewards that can be offered on mobile allows us to tailor the reward to the audience we are reaching.

We have been presented with another opportunity to reinvent surveys and how we go about collecting market research data. Let’s get it right this time around.

Melanie Courtright is vice president of research services, Research Now


11 years ago

Same sh*t different day quite frankly. Wether it's on a desktop monitor or a 3.5 inch screen in the hands of a participant it's still a clipboard. We need to junk self complete collection altogether. 'Survey' based collection methods, regardless of channel are outdated, slow and very misleading. Industry needs to embrace real technological change. Fusing freely available data sets on scale to make bigger more meaningful stories.

Like Report

11 years ago

Mobile in necessary in this busy life because nobody has spare time to visit to relative for an hour, with the help of mobile you can save time.

Like Report

11 years ago

So one lesson that Anonymous didn't learn from the transition to online is not to hype new methodologies at the expense of the old. We can improve surveys and make them even better for decision making -- Big Data won't replace them. As to Melanie's original point, it is well taken, and I look forward to reading more of her research on research about their mobile projects.

Like Report

11 years ago

Thx jeffrey! We need to think like the Brady bunch. How do we live together in a blended family and all find ourselves stronger and happier for the change.

Like Report

11 years ago

To be frank, I am astonished that we are still seeing blinkered comments like that of anonymous at the top there. New tech is not a replacement of survey's, it is simply an aid to making them more complete. I am currently working with market research, high tech and big data (or whatever you want to call it) and the fact of the matter is my proposition works best when all three are on the table together...

Like Report

11 years ago

I agree that there will always be need for appropriate research design that take into consideration research limits such as target audience, tech diffusion, time for delivery of research findings and key insights with given resources. As such technology is a tool that should be used within the context of research needs. if online panels have the representative sample required then i will use it if i have to use the cave man clipboard and hit the face to face so be it. however my research design imagination will not be limited but will be guided by reliability and validity considerations in research design.

Like Report