Mobile research – great hope or false dawn?

• Michael Alioto, vice president of Gongos Research, who kicked off the online chatter when his firm published a press release entitled Smartphone Surveys Prove Their Validity in Marketing Research.
• Lenny Murphy, editor-in-chief of the GreenBook Blog, who had fanned the flames of the debate by blogging the results of the study.
• Ray Poynter, author of The Handbook of Online & Social Media Research, who had countered with “Gongos Research have NOT proved the validity of smartphone research” and then chastised them further by issuing TARSK 17 (Things All Researchers Should Know): “What do we mean by ‘proving’ something?“.
• Reg Baker, COO of Market Strategies, who had chimed in saying, “Here we go again.”

Strohmenger did an excellent job moderating the event. But what happens when an irrepressible force meets an immovable objector, or two? Sparks flew, in an entertaining and respectful manner, as Murphy – the champion of mobile – was battered but unbowed by the objections of Poynter and Baker.

Michael Alioto
Alioto started off the debate recapping his firm’s study. As background, he said that he had encountered two schools of thought about mobile research:

1. Smart phones are strategic enhancements to online.
2. Smart phones are a different methodology that could well be the next evolutionary platform of research and quantitative analysis.

More background on the study:
Gongos Research tested a variety of research objectives on the smartphone platform, including concept evaluation, segmentation confirmation and respondent engagement. In conjunction with this, critical quantitative elements were evaluated, including scale design, survey length, open-end response feasibility, image-based stimuli, incentive levels, and complex analytical techniques, e.g. MaxDiff trade-off analysis.

Of the multiple findings, one validates the use of five-point, end-anchored scales when conducting product pricing, purchase consideration and product comparison exercises. Another shows that MaxDiff trade-off exercises can be done on the smartphone platform, allowing companies such as Best Buy to gain reliable consumer feedback in a more true-to-life environment. Finally, open-ended questions on smartphones yielded equally rich qualitative content (with smartphone responses averaging 65 characters vs. 59 characters for online).

One item that Gongos didn’t test, but would like to in the future, is how these results differ by demographic segments such as millennial generation and nonacculturated Hispanics, for instance. Alioto concluded by asking: “Is this a fifth methodology or an evolution of the online platform? With the number of smartphone platforms going through a change before our eyes, we must think about testing. The move from CATI to online was not as organised as we would like, and the smart phone revolution is coming faster than we would care to admit.” [All quotes are from my notes and should be considered paraphrases.]

Ray Poynter
Poynter said: “Had the wording of the study’s title been different than ‘proving the validity’ I would have been full of praise. We need this research. But it nets out to say: some studies in some situations work some of the time. Had it not worked, it would have meant: some studies in some situations don’t work some of the time. We need to collect more studies.

“What about the bigger issue? Is mobile going to be a niche? It will be a niche for some time to come. Most research is quantitative, and most of that is customer satisfaction, brand tracking and concept tracking. These are 20, 30 and 40-minute surveys that won’t go on to be mobile. There are 6.9 billion people on the planet and 5.9 billion mobile phones – surely we can use mobile, but we can’t use it for what we currently deliver. Will clients change what they buy? Lots of contracts are tied to a particular way of doing things.” Clients are unlikely to field true mobile surveys yet – for instance, short surveys that allow respondents to take a picture of an item they are having an issue with and perhaps record a voice annotation.

“If you look at history, why did paper go to CATI?” Poynter asked. “Because it was faster and cheaper, not because it was better. It didn’t need print rooms and the posting of questionnaires around the country. CATI then went to online not because it was better – yes, some studies are better online – but it went because it was cheaper and faster, a lot faster. Is mobile cheaper and faster and better? Let’s put better to the side as not important. It’s not cheaper – most online surveys are programmed by scripters than done by machines with incentives paid to respondents. Same with mobile. Mobile won’t be cheaper than online. Is it faster? CATI to online went from 4 weeks to 2 weeks and data collection went from 2.5 weeks to 4 days – with mobile you still need a certain time to have it in the field, as you don’t want to take the opinions of just those who answer in the afternoon.

“There you have it in my humble opinion, with humble being ironic!”

Lenny Murphy
Murphy said: “Thinking about taking the traditional quantitative research paradigm and stuffing it in the mobile channel won’t work: it won’t fit there, and that is niche. But that’s thinking about things the wrong way. We have to look at how clients embrace other types of technology. Too often, they are not using research firms for such things because research firms don’t offer these new techniques. If research firms want to earn their seat at the table and earn that prestige, they need to approach mobile differently. It’s radically different from how we think of research today. Yes, we need to find ways to accommodate traditional quantitative and qualitative, but if that is all we focus on, we will miss the opportunity.

“We have a three-to-five-year window before we are left in the dust, when everyone on the planet is using some device bigger than an iPhone and smaller than an iPad as their primary communications device. If we can’t capitalise on it, our industry will continue to be marginalised.”

Reg Baker
Baker said: “I at least agree with Michael that this is a technology that we need to pay a lot of attention to. How do we do it and how do we do it well? There is very cool stuff possible, whether quant as he talks about or some of the ethnographic studies. But I agree with Ray’s point that it is a niche. I disagree with Lenny and think it is going to be a niche for quite some time. Penetration levels are not what people make them out to be – smart phones with web access are at 10-15% of the US population depending on how you crunch the numbers. The billions of mobile phones around the world are not smart phones.

“Mobile research doesn’t solve the problem that people don’t want to talk to us anymore. Mobile will not suddenly cause people to tell us what they are thinking or doing or feeling more than other methods. That’s a serious problem outside the sliver of people willing to participate on panels. So let’s not oversell it before its time. Let’s not make the mistake we’ve made before, making evidence based on anecdotes, rather than on wide bodies of studies, as science does when it moves forward.

“The underlying sample paradigm was cast aside in the move from CATI to online. Who did we talk to and why are they important? Not only do we have lousy response rates for telephone, small percentages on panels (and small percentages of those panelists doing our surveys), but of the 600 million people on Facebook there is 1% contributing content and 8-9% commenting on that. No matter what methodology we are listening to today, we are listening to a tiny portion of the population.”

There was much more said, and you can follow the ongoing discussion on Twitter at #MRMW11.

Jeffrey Henning is chief marketing officer at Affinnova. He blogs at Innovation Evolved