Since the turn of the century, there has been an explosion in the availability of data, both inside and outside client organisations. The growth in capacity and flexibility of computing has allowed businesses to harness this information in ways that weren’t possible in the past. So now a wide range of data is available to our clients, from which they can extract insight and business intelligence – from CRM and transactional data through to point-of-sale data and radio frequency identification (RFID).

Meanwhile, the digitisation of society has created hitherto undreamed of activities and concomitant behavioural traces – social media data, such as blogs, videos, photos; mobile and location data; digital behavioural data; and e-commerce data. 

These developments have led to a fundamental change in the research industry. When I joined it in the early 1990s, we were all about designing, executing and reporting on surveys. But the explosion in data and digitisation has meant clients’ expectations have changed. Rather than designing projects to answer specific research objectives, they require us to meet broader business aims using whatever information is available and appropriate.

Market research, of course, has long been an industry of data aggregators – combining surveys, but at the same time taking our clients’ CRM and transactional data and making sense of this. 

However, in the new paradigm that big data creates, our role has changed. We now need to define, for each customer assignment, what we measure and how. What is the best way to gather each piece of information? Should we create a survey, use passive measurement, scrape the digital sphere, or create observational frameworks? Traditional surveys are just one of the ways for consumers to make their voices heard – and given the wealth of data available via other sources, it is incumbent upon us to make surveys shorter and smarter.

Furthermore, market research is built around implicit and explicit consumer frameworks. By contrast, big data is unashamedly atheoretical. We, as researchers, have adopted a wider role – that of ‘curators’: identifying, sourcing, integrating, analysing and interpreting the full range of available data to deliver insights that we would not have achieved by more traditional means. This covers the integration of sources via imputation/fusion/matching, as well as via design. 

Arguably, we have become the keepers of the total customer view – not just for that bit of their day when they are visible to a brand. We can see the bigger picture by using all available sources to understand the holistic consumer journey and the motivations that drive it. We understand the provenance of data and provide context to it. 

This offers us a huge opportunity to explore consumer insights in new ways. It enables us to combine our traditional currency of survey data with all forms of big data – to deliver truly ‘smart data’.

Corinne Moy is a Fellow of MRS and global director of marketing sciences, GfK NOP