A mobile solution

Agency and client market researchers discussed the advantages and anxieties of mobile-based research at a recent event run by Basis Research. Bronwen Morgan listened in

Robin Hilton Katherine Feres

In recent years, mobile has been hailed by many as a panacea to address a number of quantitative research issues, such as reaching younger, male respondents, improving sample representation and expanding the situations in which surveys are completed. In fact, the message put forward by the most emphatic supporters of mobile has been that the industry must switch to mobile or risk compromising quality and accuracy. 

To explore just how much weight these claims hold, Basis Research carried out an experimental study measuring the impact of a shift from PC-based surveys to a mobile-optimised approach. 

The results revealed a number of interesting trends: a switch to mobile surveys didn’t automatically attract younger men (though shorter, micro-surveys and SMS invites did increase participation among that demographic); and there was no effect of using a mobile approach on out-of-home survey completion, or indeed on survey experience. 

The results also showed that when given a choice between completing a survey on mobile or PC, the vast majority chose a PC; and, perhaps most importantly, a mobile-optimised survey design actually led to a decrease in data accuracy. 

Basis Research invited a group of client-side researchers, research agency and panel company representatives to a roundtable discussion in London to share the results of this study and reflect on the implications. A number of key themes emerged. 


Mobile is an option, not the answer

Nick Bonney: It’s about horses for courses in my view. The reality is we’re not going to get detailed, long survey capture on a mobile device; what we are going to get is second screen behaviour, slightly distracted, quick snapshot, ‘here’s how I’m feeling about something at the time’. Trying to tailor your method and your content appropriate to the device feels like the key thing.

Katherine Feres: It’s the responsibility of researchers, whichever side of the fence you sit on, to choose the right methodology. If you’re trying to get in-the-moment feedback that points me more towards going down a qual route, as opposed to expecting that, on a mass scale, you’re going to be capturing people in the moment. 

Judith Welford: Young people are happy to use mobiles for different sorts of activities. They don’t necessarily have to be on the move to use their mobile to purchase things, for example. So the question is: what do we want from them? Is it a short interaction or is it something more? It should be the agency’s responsibility to advise the client.

Aaron Simmons: I think there is a distinction though, between making this accessible to everybody and using mobile as a methodology, as a stand-alone. Where there may be an opportunity to do some diary type work via mobiles – be that in-app or web browser – it’s a stand-alone methodology that does have limitations. Opening that up and letting people participate in a way that is more akin to their own browsing behaviour – that’s a separate approach; that’s more device agnostic.


Mobile-specific sample

Richard Bowman: The fix is to stop using the term ‘online research’ and start talking about ‘PC research’ and ‘mobile device research’. I would happily buy PC sample at one price and know that I could probably stretch to a 20-minute survey; and pay a bit more for mobile panel knowing that somebody had been recruited through mobile, trained on a mobile device, and only given mobile surveys.

Rune Mortensen: If you’re getting a more representative sample, who feel more engaged and are enjoying it more, your data is going to be better quality. 

Simmons: The problem with that is that quality is in the eye of the beholder, and it can change. We tried to define quality and gave up, because it depends on whether it’s the service offering, whether it’s down to the data you’ve got, whether it’s the respondent… because we can have a good respondent and bad data, or good data and a bad respondent. The survey plays a role in that as well.

Mobile-specific design

Bonney: Historically, we’ve not been that brave as an industry: trying to design the same thing and bung it on a mobile device, rather than reinventing it to work on mobile. I recall when the first flash tools were around, we did parallel tests on those we were using, and we were advising clients not to use half of them, because the data came out wrong. We can’t just think: ‘Right, it’s mobile, we’ve got to make this loads better.’ It’s actually the opposite: you need to strip it down, make it simple. It’s also about looking at what’s good and easy for both the researcher and the respondent and getting the best of both of worlds.

Robin Hilton: Most surveys we design are agnostic; they automatically repurpose questions to be suitable for mobile. But, even when we say the survey isn’t suitable for mobile, there’s still a significant percentage of people taking it on their mobile. They just prefer it that way. So, we’ve gone down the route of making the surveys more user-friendly and engaging. Panellists filling in multiple surveys get bored of seeing the same thing. Also, if the surveys weren’t getting longer and longer, you probably wouldn’t need to put as much design and user experience into it.


Paul Nesbitt: I recall seeing a study where a standard survey had been tested, then another version with enhanced UX, and a third with some added gamification. The gamification survey didn’t perform any better than the UX version. But, when the design was tweaked, better scores in terms of overall satisfaction were seen and participants gave more responses to multi-code questions compared with a ‘standard’ survey. The question is: ‘has the industry got the gamification and UX surveys right?’

Mark Bagnall: Even if respondents are saying: ‘I don’t want it to just look like boxes’, it doesn’t mean it’s going to be more accurate because you’ve now got lots of widgets on. So they might not like it [a plainer survey], but is it actually better for accuracy?

Feres: There are buzzwords of gamification and whatever else, but what I’m looking for from a mobile survey when I get links through, is: ‘Does it work? Can I click on the buttons and move through the questions?’ I’ve never had a conversation around gamifying my survey, because my conversations are around: ‘I can’t actually progress from question six to seven right now’.

Bowman: I’ve been looking at data that shows that 50% of panellists on one panel I use don’t sign up via a PC – they’ve seen a digital advert or they’ve received an email – but around 80% across markets are taking surveys on PC. They’ve come onto a panel and then, a day later, they’re getting a 20-minute survey that’s a bit crappy on PC and terrible on mobile – of course they’re going to disappear. 

We hope you enjoyed this article.
Research Live is published by MRS.

The Market Research Society (MRS) exists to promote and protect the research sector, showcasing how research delivers impact for businesses and government.

Members of MRS enjoy many benefits including tailoured policy guidance, discounts on training and conferences, and access to member-only content.

For example, there's an archive of winning case studies from over a decade of MRS Awards.

Find out more about the benefits of joining MRS here.

0 Comments

Display name

Email

Join the discussion

Newsletter
Stay connected with the latest insights and trends...
Sign Up
Latest From MRS

Our latest training courses

Our new 2025 training programme is now launched as part of the development offered within the MRS Global Insight Academy

See all training

Specialist conferences

Our one-day conferences cover topics including CX and UX, Semiotics, B2B, Finance, AI and Leaders' Forums.

See all conferences

MRS reports on AI

MRS has published a three-part series on how generative AI is impacting the research sector, including synthetic respondents and challenges to adoption.

See the reports

Progress faster...
with MRS 
membership

Mentoring

CPD/recognition

Webinars

Codeline

Discounts