Mobile survey design still hampering research

The companies collated and aggregated three years’ worth of panel data to identify the response patterns between devices by age and country in seven key markets.

The 2018 data found that almost a third ( 30%) of surveys globally restrict respondents to a desktop experience only, and 20-25% of surveys did not allow phones at all.

While the number of surveys that have been started and completed using a mobile device has increased significantly – up by 10 points over the past year – a respondent is still more likely to abandon a survey because of poor mobile optimisation compared to if they were taking the survey on their desktop.

“Research design that does not fully result in a mobile-friendly experience is often the culprit behind resulting non-representative audiences and feasibility challenges,” said Courtney Williams, executive director of quality at Lucid, discussing the research findings during a recent MRS webinar. “From our perspective, the goal is parity with desktop devices, where respondents find it as realistic to successfully complete a survey experience regardless of what device they choose to take it on.”

The analysis also found that while females are persistently higher in terms of propensity to use mobile devices, the gender gap started to close slightly between 2017 and 2018, with a 6% increase among males’ use of mobile devices for surveys, versus a 4.8% increase for females.

However, regardless of age or gender, there is a clear pattern that if a respondent starts a survey on a mobile device they are less likely to complete it, compared to survey users of other devices.

For the first time this year, the researchers also looked at the performance of trackers versus ad hoc surveys, hypothesising that trackers would have more legacy design implications preventing mobile compatibility, but the data demonstrated that there aren’t significant differences between survey types.

There are a number of indirect costs associated with not having an equal share of surveys completed on a mobile device compared to PC or laptop, said Justin Bennett, vice-president, global sampling, Toluna.“From a panel company’s perspective it’s higher attrition and more throughput to generate a given number of completed interviews. There are also hidden costs in terms of added recruitment costs to replenish panels and perhaps added administrative and sample costs.”

Pete Cape, director of global knowledge, Dynata, said: “Twentyfive per cent of all survey opportunities are not available to mobile devices, and it’s not the fault of trackers. So it’s time to put your hands up and say ‘are my surveys mobile friendly or not?’ The issue is well understood now – this is not a phenomenon that’s going away, it’s not safe to ignore – it has to be dealt with.”

In terms of improving usability, mobile surveys should have bigger functional objects and buttons to make surveys easier to use, fewer images and fewer words.

Grids also pose challenges for researchers, as does NPS, because it’s an 11-point horizontal scale. A number of projects are looking into alternatives to the grid format, e.g. research from Dynata, FocusVision and MaritzCX. Testing has found that a carousel format offers the best experience, rather than a standard grid.

“The only way you’re ever going to know what the surveys are like on a mobile device is to test them on a mobile device,” said Cape.

Discussing the challenge of more complex survey design such as conjoint, Chris Stevens, chief quality officer at Kantar Profiles, said: “The respondent wants to take surveys on these devices, and we’ve got to adapt to that setup. If that does mean a complete redesign of conjoint analysis, maybe that’s what has to happen and we have to do things in different ways.”