An extra life for online surveys

On a scale of one to 10, how would you rate the entertainment value of most online surveys? Next, using the same scale, how would you rate your level of engagement with the previous question?

We’re not expecting the results to be good, but stick with us – things may be about to get better. That’s the message coming out of the research industry as a small but growing band of professionals set to work on reappraising traditional survey design to come up with something altogether more… fun.

It’s called gamification, and it’s the application of gameplay mechanics to all manner of things not typically thought of as entertaining. It’s about “making things that are normally boring – such as filling out surveys, shopping, or visiting websites, more interesting,” wrote Dean Takahashi, a VentureBeat writer covering the announcement of the first-ever Gamification Summit, which took place this January.

Loading… please wait
Surveys weren’t specifically on the agenda at that San Francisco event, but over in London gamifying surveys has been occupying the thoughts of Jon Puleston, senior director of GMI Interactive. It has been on his mind since at least 2006, when his company, then known as Media Intelligence, wowed GMI with its virtual store shelf technology. The research software maker later bought Puleston’s firm.

“The multimedia environment has forced us to up our game,” said Rob Monster, GMI’s CEO at the time. But by and large, in the intervening years, the industry has failed to do so. Radio buttons, grids, drop-down menus – all the things that were expected to be eliminated in a Flash (thanks to Adobe’s rich-media creation platform of the same name) are still the rule with surveys rather than the exception. The interactive and visually appealing future we were promised has yet to materialise.

Why? “It’s been a technological hurdle, more than an emotional one,” says Puleston. When Media Intelligence was bought by GMI it was a bespoke design house staffed by Flash developers. “We’ve spent the last three or four years building technology to industrialise the techniques,” Puleston says. “What we’ve been trying to do is create a piece of technology that doesn’t require you to have Flash developers to build interactive surveys.”

Puleston says GMI has now reached the point where it can deliver interactive surveys within the same timeframe as an HTML survey. But gamification is about more than mere inputs – done properly, it requires a fundamental reframing of the entire survey experience.

Press start
First things first: what is a game? Research asked Alex Cullum, a design director with videogame publisher Disney Interactive. He’s worked on popular titles like Populous 3 and Syndicate Wars and more recently the racing games Pure and Split/Second.

He defines a game as “an interactive activity with a pre-defined set of rules, where the objective is to entertain”. Asked to name an example of game design at its purest, he cites Space Invaders. Game design involves three things, Cullum says:

• high-level design: that is, a premise or narrative setting that makes the player want to play by offering fantasy fulfilment
• medium-level design: the incentive to keep playing from minute-to-minute – the high-score table in Space Invaders’ case
• low-level design: the second-by-second feedback to the players’ inputs that reward them and make them feel good – the sound of a laser firing, the explosion of an enemy and the shower of particles

At GMI, Puleston and his team have experimented with adding some of these elements to surveys. They’ve added the sound of a gunshot to accompany the click of a mouse. Mix that with the company’s “opinion showers” – a group of words moving and swirling about the screen – and that’s about as close to a Space Invaders-esque shoot-‘em-up survey as you’re likely to get.

Adding scoring mechanics to surveys is also possible, Puleston says. One way is to turn a survey into a sort of psychometric test where respondents’ answers are used to determine what type of person they are. Puleston also demos a more traditional quiz-game approach, which sees a survey about the R&B singer Beyoncé segue into a song quiz to test how dedicated a fan the respondent actually is (Research scored 4 out of 18 ).

As videogames have advanced in recent years to create deeper, more realistic experiences, they have moved away from the high-score tables of old, yet they still retain an element of scoring through a system of achievements – where badges and points are awarded for tasks that have been completed in-game.

Cullum says achievements have been very successful in encouraging people to play more games and to play the ones they own more comprehensively. “It is essentially an example of medium-level design – the minute-to-minute rewards and incentives – which has been taken out of the game world,” he says.

It wouldn’t take much for panel companies to add some form of achievements to their community management platform – for example, awarding badges for the number of surveys completed, or for introducing new members to the panel. But this approach to gamification is sometimes derisively referred to as ‘badgification’: sticking a badge on something without really changing the underlying experience.

In a similar fashion, scoring systems and sound effects – however novel they are in the survey context – could be introduced without fundamentally altering the way respondents are asked questions and the way in which they respond. It is here research has most to gain from gamification.

Game on
“The industry is dogged with this sense that you have to word survey questions in a particular way,” says Puleston. “You have this whole language of surveys – ‘how much do you disagree with this statement on a scale of one to 10’, that sort of thing. It just turns people off. It’s a cliché. Imagine Michael Parkinson having to conduct interviews using only survey questions. How far do you think he’d get in terms of insight and feedback?”

Puleston wants the research industry to reconsider the way questions are phrased and how they could be changed to make them more game-like. “We did an experiment,” he says, “where we asked people to name their favourite meal, but one group of people were instead asked to imagine they were on Death Row and to say what their last meal would be.” The result, Puleston says, was “an order-of-magnitude improvement in the quality, depth of thought and effort” that people put into answering. The Death Row group spent an average of two minutes on the task, the control group 15 seconds.

In another experiment, working with Sony, GMI tried an alternative approach to asking people to evaluate which music artists they liked. One group was put through a traditional survey approach, another was asked to play the role of a radio station owner and they were asked to build a playlist. On average the first group evaluated 82 artists before giving up; the radio station owners got through 146.

Time-based challenges have also proved successful, Puleston says. Rather than asking respondents to name all the foods they like to eat, they were told to imagine they were about to go food shopping to stock up on everything they’d want to eat for an entire year, but they only had two minutes to write out a shopping list. “What that seems to do is unlock some bit of mental processing – an enthusiasm enzyme – that generates a huge amount of activity among respondents, so instead of listing five or six items, they list 30 products,” says Puleston.

Asking questions in different ways clearly has an impact on the data outputs. But, in data terms, does different necessarily mean better? “I can only see that the more engaged people’s minds are, and the more enthusiastic they are to do a task, the more thoughtful the feedback they give,” says Puleston. “Yes, the data is different, but if it was me I’d want that difference. I think that it’s a symptom of improved quality data.”

Tests by BrainJuicer of its own, now well-established use of role-playing in research go some way towards supporting this view. The online research agency uses predictive markets in new product development research, asking respondents to play the part of a venture capitalist with money to invest in the product ideas they think will be most successful. Comparisons with monadic product testing show predictive markets to be equally as good in identifying the top-quartile winners, but the approach sets itself apart in discriminating much more between the very good and the very average product concepts.

Level two
But quality of data has as much to do with the way respondents can give feedback as it does with they way they are asked to do so. Straightlining through grids of answers, for example, is symptomatic both of the respondents’ disengagement with the questions, and of the format in which they have been asked to respond.

Conquest, a London-based agency, realised this in developing its range of brand evaluation tools, which did away with grids and numeric ratings scales in favour of animated avatars which could be moved around in various environments to denote a respondent’s feelings. For instance, feelings of excitement towards a product might be expressed by raising an avatar off the ground, while respondents could show how comfortable they are with a brand by moving two avatars closer together on a sofa.

The survey-creation technology that Puleston and his team have spent the best part of four years working on, Qstudio, has 15 different ways of asking a grid question without resorting to a grid format. These include drag-and-drop formats that might ask the respondent to place flags along a scale or pin pictures to a target, as well as a variety of different slider-based inputs.

Like GMI, Annik Technology Services appears to have overcome the technological hurdle preventing wider uptake of interactive survey formats. Its SurveyFlashTools, a library of plug-in customisable Flash objects, won the 2010 MRS/ASC Award for outstanding innovation in research software. In conferring the award the judging panel said: “In an industry that is increasingly challenged by declining response rates and respondent engagement, SurveyFlashTools simplifies a process that has, up to now, been largely restricted to technical specialists.”

Continue/Quit?
With the right tools at the industry’s disposal and a willingness to re-think the way questions are asked, survey gamification stands a chance of succeeding. But while there are plenty of upsides, making the survey experience more game-like could bring unwanted consequences. A piecemeal approach – including only a few gamified questions among a host of more traditional ones – will only serve “to highlight how boring three-quarters of the questions are,” says Puleston. And incorporating game mechanics that are too engaging in themselves risks distracting from the survey goals, warns Jamie Madigan, a personnel psychologist for the US federal government and author of the Psychology of Video Games blog.

“There’s research to show that some people are more likely to get into a state of psychological flow – they get into a zone where the difficulty of the game is perfectly matched to their ability and they do things without really thinking about them,” says Madigan.

“In the context of a survey, if you break that flow by suddenly asking people to do something very different to what they were doing before, people might be apt to do whatever it takes to get past that and get back to the fun stuff – they might give you dirty data or try to skip sections of the survey.”

Different types of people will also have different opinions of what constitutes fun – some will gain enjoyment from attempts to bend or break the imposed rules, says Tom Ewing, social media knowledge leader for Kantar Operations. Of course most will play fair, but common to nearly all gamers is a desire to experience a sense of autonomy and mastery over a game world – both of which sit uneasily with the linear, regimented nature of most survey research projects and their requirements for respondents to answer those questions, or complete those tasks, in that order.

Do researchers have to be willing to cede control and give more autonomy to respondents in order to create truly game-like, interactive experiences? Cullum, the Disney design director, suggests not.

“Ultimately, it is important for the designer to control how a game progresses so that they can maintain the challenge-and-reward curve. The real trick is to hide that control from the player. The bad games are the ones that leave the player thinking, ‘What does the designer want me to do next?’ The good ones are those where the player thinks, ‘What do I want to do next?’”

And the great ones? “Those are the games where the designer’s and the player’s desires are the same,” says Cullum.

Gamification is “an exciting, risky opportunity for research,” according to Tom Ewing, and it’s hard to disagree. One of the most promising things about it, he says, is that it puts respondent engagement in the spotlight. And if there’s nothing else the industry learns from games design it should be this: “You are creating an experience for someone else,” says Cullum. “There is a more important active participant involved than just the designer… the player always has to be the most important person in the experience.”