Second screening: how data science finds better participants

“The trouble with market research is that people don't think how they feel, they don't say what they think, and they don't do what they say,” David Ogilvy famously once said.

Healthy scepticism is at the heart of what we do. This doesn’t mean that we don’t believe the people that take part in our studies, it just means that a critical part of our job is interpreting not just what they say (or don't say), but how it’s said.

We also know that it’s not just what people say in focus groups and interviews that we need to treat with an open mind, it’s also what they say in the screening process. It’s hardly news to say that finding genuine, viable people to include in our studies is tricky, but this wasn't as much of an issue in the past. Six to eight people who buy washing powder a few times a year are easy to find, and it won’t be the end of the world if they earn a little over or under the ideal amount. Recruiters were always wary of answers seeming too-good-to-be-true, and would screen out ‘professional respondents’.

But as targeted marketing becomes more sophisticated, there’s increasing pressure from clients to find and research increasingly niche audiences. Now we’re looking for the man who goes on holiday by himself more than three times a year (and is going away in the next month); owners of three (but not four) cats; and the woman who plays first-person shooter video games for more than 10 hours a week. It’s very challenging for even the most experienced recruiters to truly verify these sorts of profiles. They, and we, have little option but to trust what people have told us and hope that a majority of good fits would average out a couple of bad ones.

As our clients request to speak to more rarefied and specific people, we've found what we now think is a good solution.

At Flamingo, we have the ability to gather data on people’s digital behaviour and geographical movements. With their permission, we can, among myriad other things, track their video game library or YouTube consumption, GPS and search history. We use this information to build a much more accurate picture of the people taking part in our studies, their movements and their behaviour. Because as we know, human memory is fallible; we don’t always remember our exact sequence of behaviours, and where we were when they took place.

But, as we recently discovered, this also allows us to quickly detect when someone is deliberately not telling the truth about their behaviour, which means it’s a great way of ensuring that our work isn’t compromised.

For example, on a recent project, we were looking to track travellers for a week before, during and after a holiday abroad to get a sense of how their behaviour changed. The criteria to take part in the study was that they should be travelling for leisure more than three times a year, and were going on holiday in the next month.

Of the 16 people we recruited for this project, a quarter were revealed in the digital tracking to be inappropriate for the study. One person had told our recruiters they were going away on one holiday, but the data suggested they worked for an airline (which was one of our listed exclusions); they were using apps specific to cabin crew and carrying out multiple single-day trips. Another didn’t go on holiday at all, but didn’t let us know. But we had the geolocation data to prove they hadn’t left their home city and quickly replaced them in our study.

Another project required us to find participants that were experts on a specific genre of video game. We were looking for players of a very specific combination of titles, but this didn't stop other gamers from applying to our project.

Many claimed to have spent hundreds of hours playing these specific games, but their data didn't back this up. In some cases, we could even see them searching on their phones for the answers to the questions we were asking them. We’re often asked if people adapt their digital behaviour when they know they’re being tracked; this behaviour clearly illustrates how quickly people forget that we can see what they’re doing. Across six cities, we screened out over half of the people who had claimed to fit our criteria.

It’s difficult to overstate the issue of mis-recruiting. Having unsuitable people take part in our studies at best warps results, and at worst calls the whole project into question. The shift to micro audiences and hyper-specific recruits only magnifies this; the fewer people we involve, the more onus there is to make sure each one of them is perfect.

Working closely with our recruitment partners, digital tracking offers a second screening stage on projects where finding exactly the right people is paramount. With the industry evolving, data-fuelled targeting needs a data-led solution.

Lydia Crudge is senior strategist and Lucas Galan is head of digital forensics at Flamingo