OPINION8 July 2011

Mobile research – great hope or false dawn?


The mobile phone as research tool has plenty of enthusiastic supporters – and as many detractors. Affinnova’s Jeffrey Henning listened in as the two sides squared off in a web debate this week assessing the cell’s place in the grand scheme of things.

Strohmenger did an excellent job moderating the event. But what happens when an irrepressible force meets an immovable objector, or two? Sparks flew, in an entertaining and respectful manner, as Murphy – the champion of mobile – was battered but unbowed by the objections of Poynter and Baker.

Michael Alioto
Alioto started off the debate recapping his firm’s study. As background, he said that he had encountered two schools of thought about mobile research:

  1. Smart phones are strategic enhancements to online.
  2. Smart phones are a different methodology that could well be the next evolutionary platform of research and quantitative analysis.

More background on the study:
Gongos Research tested a variety of research objectives on the smartphone platform, including concept evaluation, segmentation confirmation and respondent engagement. In conjunction with this, critical quantitative elements were evaluated, including scale design, survey length, open-end response feasibility, image-based stimuli, incentive levels, and complex analytical techniques, e.g. MaxDiff trade-off analysis.

Of the multiple findings, one validates the use of five-point, end-anchored scales when conducting product pricing, purchase consideration and product comparison exercises. Another shows that MaxDiff trade-off exercises can be done on the smartphone platform, allowing companies such as Best Buy to gain reliable consumer feedback in a more true-to-life environment. Finally, open-ended questions on smartphones yielded equally rich qualitative content (with smartphone responses averaging 65 characters vs. 59 characters for online).

One item that Gongos didn’t test, but would like to in the future, is how these results differ by demographic segments such as millennial generation and nonacculturated Hispanics, for instance. Alioto concluded by asking: “Is this a fifth methodology or an evolution of the online platform? With the number of smartphone platforms going through a change before our eyes, we must think about testing. The move from CATI to online was not as organised as we would like, and the smart phone revolution is coming faster than we would care to admit.” [All quotes are from my notes and should be considered paraphrases.]

Ray Poynter
Poynter said: “Had the wording of the study’s title been different than ‘proving the validity’ I would have been full of praise. We need this research. But it nets out to say: some studies in some situations work some of the time. Had it not worked, it would have meant: some studies in some situations don’t work some of the time. We need to collect more studies.

“What about the bigger issue? Is mobile going to be a niche? It will be a niche for some time to come. Most research is quantitative, and most of that is customer satisfaction, brand tracking and concept tracking. These are 20, 30 and 40-minute surveys that won’t go on to be mobile. There are 6.9 billion people on the planet and 5.9 billion mobile phones – surely we can use mobile, but we can’t use it for what we currently deliver. Will clients change what they buy? Lots of contracts are tied to a particular way of doing things.” Clients are unlikely to field true mobile surveys yet – for instance, short surveys that allow respondents to take a picture of an item they are having an issue with and perhaps record a voice annotation.

“If you look at history, why did paper go to CATI?” Poynter asked. “Because it was faster and cheaper, not because it was better. It didn’t need print rooms and the posting of questionnaires around the country. CATI then went to online not because it was better – yes, some studies are better online – but it went because it was cheaper and faster, a lot faster. Is mobile cheaper and faster and better? Let’s put better to the side as not important. It’s not cheaper – most online surveys are programmed by scripters than done by machines with incentives paid to respondents. Same with mobile. Mobile won’t be cheaper than online. Is it faster? CATI to online went from 4 weeks to 2 weeks and data collection went from 2.5 weeks to 4 days – with mobile you still need a certain time to have it in the field, as you don’t want to take the opinions of just those who answer in the afternoon.

“There you have it in my humble opinion, with humble being ironic!”

Lenny Murphy
Murphy said: “Thinking about taking the traditional quantitative research paradigm and stuffing it in the mobile channel won’t work: it won’t fit there, and that is niche. But that’s thinking about things the wrong way. We have to look at how clients embrace other types of technology. Too often, they are not using research firms for such things because research firms don’t offer these new techniques. If research firms want to earn their seat at the table and earn that prestige, they need to approach mobile differently. It’s radically different from how we think of research today. Yes, we need to find ways to accommodate traditional quantitative and qualitative, but if that is all we focus on, we will miss the opportunity.

“We have a three-to-five-year window before we are left in the dust, when everyone on the planet is using some device bigger than an iPhone and smaller than an iPad as their primary communications device. If we can’t capitalise on it, our industry will continue to be marginalised.”

Reg Baker
Baker said: “I at least agree with Michael that this is a technology that we need to pay a lot of attention to. How do we do it and how do we do it well? There is very cool stuff possible, whether quant as he talks about or some of the ethnographic studies. But I agree with Ray’s point that it is a niche. I disagree with Lenny and think it is going to be a niche for quite some time. Penetration levels are not what people make them out to be – smart phones with web access are at 10-15% of the US population depending on how you crunch the numbers. The billions of mobile phones around the world are not smart phones.

“Mobile research doesn’t solve the problem that people don’t want to talk to us anymore. Mobile will not suddenly cause people to tell us what they are thinking or doing or feeling more than other methods. That’s a serious problem outside the sliver of people willing to participate on panels. So let’s not oversell it before its time. Let’s not make the mistake we’ve made before, making evidence based on anecdotes, rather than on wide bodies of studies, as science does when it moves forward.

“The underlying sample paradigm was cast aside in the move from CATI to online. Who did we talk to and why are they important? Not only do we have lousy response rates for telephone, small percentages on panels (and small percentages of those panelists doing our surveys), but of the 600 million people on Facebook there is 1% contributing content and 8-9% commenting on that. No matter what methodology we are listening to today, we are listening to a tiny portion of the population.”

There was much more said, and you can follow the ongoing discussion on Twitter at #MRMW11.

Jeffrey Henning is chief marketing officer at Affinnova. He blogs at Innovation Evolved


13 years ago

Disappointed no one covered mobile ethnography, Which is cheaper and better when used with mobile devices, Speaking as someone with 15 years of experience in participant observation behind me. Wanna debate this?

Like Report

13 years ago

Thanks for the thorough recap Jeffrey. Siamack, I too was hoping to hear more about mobile as it relates to Qualitative research. At the MRMW11 event in Atlanta, I'll be sharing my experiences using mobile qualitatively and looking forward to some stimulating discussions!

Like Report

13 years ago

Here's one example of a mobile ethnography covered on The NewQual blog the day after this year's Super Bowl: http://www.newqualitative.org/blog/7-learnings-from-the-super-bowl-xlv-2011-mobile-ethnography/

Like Report

13 years ago

Siamack, I apologize -- I was only able to recap the opening comments. There were discussions about self-ethnography through mobile apps later on -- it's definitely something that the speakers found very promising. But it wasn't the main focus of the discussion.

Like Report