Welcome to the party

The only thing we need to know about customers is what they bought. You remember this. It was a decade ago, just around the first time you heard the phrase ‘big data’, the unassailable future of marketing.

We were invited to see fireworks light up London. So bright and dazzling, they would surely fry market research in brilliant sparkly glory. The digital economy would soon rid itself of traditional customer insight. Big data would produce a seamless insight machine capable of drilling into the slightest nuance – insight for the omnipotent. Doomsayers ran amok.

We pulled our coats on and trundled down to the river to accept our fate, huddled, turned skyward into the night. Ready to be blown away by big data’s opening salvo, we waited for the music to start. One year away from 2020, craning necks now sore, we realise it was a damp squib that went off years ago. Let’s hope the pubs are still open.

In the meantime, big data delivers the spectacular; particularly with passive measurement, including mind-blowing programmes in security, the environment, telecoms, transport, governance, and more recently, healthcare. The joke making its rounds is that Hadoop clusters, the engines of big data architecture, could be a cause of global warming.

I’m not saying it’s a good joke but do look around – big data has hardly held a rave inside customer insight. Beyond the cordon of digital logs, programmatic advertising, transactional, and loyalty card data, there is collective rejection of broader or permanent surveillance for commercial interests.

In this space, big data did not land the touted game changer; we have yet to stumble across anything that big data has achieved that small data could not. The underlying reason: small data is plenty. We’ve known this for aeons. Unless you are a nuclear scientist or programming my autonomous car, you really don’t need to be more than 95% confident of anything.

Survey data is far from perfect. Ashamed of survey bias, we are too quick to discount the relative strength of artfully designed, to the point questioning. We struggle to square away recall, needs, values, and intent with actual behaviour and market share. Big data has the same problem; it just isn’t broadcast.

Similarly, we forget that we are accustomed to asking a lot more of our small survey data than is reasonable, then take for granted how well it withstands the weight of scrutiny. Big data on the other hand is temperamental, narrow, and noisy, often arriving to the party with unsolvable conceptual failings of its own.

For these reasons, big data is the epitome of hype. Yet in 2018, we still suffered the ridiculous argument that traditional market research is a dying flame, simply because there is more data in the world.

It is a dangerously self-fulfilling argument, aimed at making market research feel uninvited, cold and small. Notice that this is without fail, the jibe of those who are not given the task of routinely extracting insight from data. Unchallenged, it will linger to stifle engagement and reroute investment. It’s true, there is inconceivably more data in the world, but it is universally accepted that almost all of it is irrelevant, siloed, classified, partial or unusable.

Therefore, there will be no revolution in our vibrant industry, no hammer to fall because we’re always evolving. For the past two years, market researchers have welcomed big data to the party, integrating it into research programmes because it helps us where survey data cannot, offering unparalleled granularity – not because we indulge the idea that it is superior or magical.

This winter, our party is rocking and our hearth glows warm. There was never a need to throw more data on.