Lindsay Forster, CEO of data and insights firm Shepper, discusses the importance of human-based data collection amid an industry under pressure.

Retail data abstract image

We are creating more data than ever before, and increasingly complex ways to analyse it. But good old human-based data collection is still immensely important -and is finally being disrupted.

We all carry around devices that generate reams of automated data. This includes deeply personal things – our search history, the steps we are taking every day – and things of more broad interest, like how bad the traffic is on our route to work.

Automated data “messy’ and “unreliable”

This endless sea of data can be useful, but its automated nature can make it messy and unreliable. What your phone might think was you taking a brisk walk could instead have been your phone sitting on top of the laundry machine. Your ultra-personalised search history might have been your toddler mashing their fingers on your phone.

That’s why human-based manual data collection remains extremely important. Sometimes you just need an actual intelligent human to go and have a look at something, and report back on it in a systemised way. This is especially true for market research: It’s hard to imagine a robot mystery shopper getting a typical experience in store.

Human collection market research needs revamping

But market research’s reliance on human collection has also caused it to stagnate. As the world has changed around it many research firms are still sending the same people out during 9-5 hours, producing the same weekly reports they’ve been doing for decades.

At Shepper, the company I lead, we’re looking to change that.

Importance of real-time market research

We pay tens of thousands of people to do real-time market research. This can cover a wide range of activities: from taking photos of shelves for a brand that wants to make sure their brand is being stocked correctly to asking for a gin and tonic to see what the “house” gin is.

These reports do not take weeks, but hours or even minutes. This kind of speed can allow our customers to quickly test hypotheses, making decisions based on the world exactly as it is, not how it was last week.

The huge number of people we have collecting data for us – we call them Shepherds – means firms can get insights from all over the UK for us, not just from London and Leeds. (We are a global company, but largely focus on the UK currently.)

This data flows in at huge scale, but not from walking smartphones. Our Shepherds are real people who are usually collecting data in the communities they live in.

They aren’t strangers attempting to look normal in the supermarket, but regulars, who often pair data collection with their normal errands. The human element remains key, and we train many of our Shepherds to make sure they are getting the data that is relevant.

Data at scale useful for running experiments

Data collection at scale is useful not just for checking that everything is running normally, but for running experiments.

This already happens in the digital realm a lot with A/B testing. News websites might use two versions of a headline, with one segment of the audience seeing one version and another the other. The headline that attracts the most readers is obvious and can be rolled out for all users.

Real-time data lets you do speedy A/B testing in the real world too.

For example, you can use one display location in one store and another somewhere else, then get a Shepherd to see which location is resulting in more products leaving shelves.

Before the day is out you can have a clear idea of the best location to roll out to the wider market. And this doesn’t have to be limited to two options – “A/B testing” can turn in to “A/B/C/D/ testing” easily.

Data collection under pressure

Data collection as a wider field is under pressure. While it is easier than ever to collect automated data, human-based data collection is in many ways getting harder.

Polling companies, for example, find it far harder to get a representative sample to pick up the phone – leaving them with online-based surveys that by their nature skew towards a certain population. But the human touch remains as key as ever.