Adapt and transform: How technology is changing the insights process

x Sponsored content on Research Live and in Impact magazine is editorially independent.
Find out more about advertising and sponsorship.

‘Better, faster, cheaper – you can pick two.’ It’s a well-worn belief that, in the world of business, good and fast come at a cost.

Yet when Martin Sorrell, formerly chief executive of global advertising behemoth WPP, founded his own venture, S4 Capital, he proclaimed that it is indeed possible to achieve all three today – that technology has opened up a realm of previously inaccessible opportunities for marketing.

Can it also be the case for market research, in which accuracy is critical, but where speed and cost are becoming increasingly – in some cases equally – critical considerations?

Technology is driving new efficiencies and a scale not witnessed previously in research, and some of this automation is likely to significantly impact how the industry operates in the future. In some corners, it’s already happening.

Change is afoot in online research; do-it-yourself (DIY) tools and services have given clients new means of accessing insight, and data collection is undergoing transformation in how samples are sourced.

Programmatic technology allows for the buying of sample to be automated – for research buyers to access audiences at a larger scale at the click of a button through survey marketplaces, bypassing traditional processes.

Curation or technology?

Zappi was founded with the aim of automating the processes behind market research. Wojtek Kokoszka, chief operating officer at the company, believes technology can tackle sample challenges in a way that hasn’t yet been widely understood.

“For a long time, researchers – both client-side and suppliers – have had the conventional wisdom that sample has to be exceptionally carefully curated,” says Kokoszka. “The fear of the researcher has always been ‘if we don’t get the sample exactly right and consistent, then you can’t really trust the data, because it might just be slightly different ways of answering surveys and, therefore, you’re introducing a distortion or noise, rather than a signal of what’s changing’. Technology removes that problem; I just think the research industry hasn’t understood that yet.”

The reason for this, he argues, is that technology can allow the researcher to see how different sample streams from different sources react. “You can see how loyalty card programmes in the US on the west coast differ slightly from loyalty card programme respondents on the east coast. But you need very powerful, quick-reaction technology to be able to spot that on the fly and then make the correction to deal with that distortion.”

Kokoszka refers to the financial markets and trading, where technology is executed in big volumes at very high speeds – micro-seconds.

“Research doesn’t deal in micro-seconds. Real-time research means today, not a micro-second right now.”

However, he believes that is where the technology is heading. “That’s how the industry will change, if I look forward four years rather than one year. I think that consistency in sample will become a thing governed by technology, not by curation.” Kokoszka also predicts that automation will bring insight back up to the C-level within organisations.

“Before the internet, the only data a chief marketing officer and chief executive had about customers came from the market research function, so it was absolutely integral into major business decisions,” he says.

“Now, I challenge any head of insight to tell me that they spend any time with the global chief executive. I think automation re-elevates them up.”

He envisages a world where the head of insight could sit with the chief growth officer and chief executive, looking through data on an iPad that is being recalculated while they are having a strategic discussion.

“That’s never been possible before. So I think automation is very powerful to make insights people more important.”

Changing consumer behaviour

Greg Dunbar, executive vice-president of enterprise solutions at Cint, joined the company five years ago. For the first couple of years, “more than half” of his time was spent convincing some customers that change was coming and they needed to be ready to adapt. However, in the past three years, he says, that has changed.

“I no longer need to have that conversation. Customers fully recognise that change is happening and that they need to understand how to capitalise on that, not just cope with it.”

The shift is driven by changes in consumer behaviour, says Dunbar. The way people interact with brands and technology has changed in the past few years – and then again in the past 18 months during the pandemic.

While the impact of the latter is still yet to be fully understood, it’s likely to have accelerated the shift towards personalisation and fragmentation of people’s attention.

“Consumers are using multiple channels and they expect everything to be on-demand, personalised, for their privacy to be protected, and for products and services to be tailored to their needs, so brands have had to adapt,” says Dunbar.

Marketing as a discipline has become considerably more data-led in recent years, with marketers having access to automated digital tools in marketing and advertising technology – while customer experience management (CXM) is currently one of the biggest areas in technology investment.

Market research, while it has always played a “really critical role” in informing decision-making, remains “very disconnected and very analogue, for the most part”, says Dunbar.

“Research has been digitised with online research, but the operations and processes used to fulfil that research, particularly of data collection, are mostly still very analogue. There is lots of manual human intervention and there are lots of human beings doing very repetitive tasks.

“The fieldwork and the recruitment of respondents, the sampling and fieldwork process – that is fundamentally where things tend to fall apart. There’s an enormous bottleneck in the system. If that doesn’t work or isn’t as optimised as it should be, the entire process falls to the lowest common denominator.”

The need for speed

It has been a challenge for the industry for quite some time: clients want speedier access to insight, a trend that has only accelerated as a result of the economic and behavioural uncertainties caused by the pandemic.

“If we’re looking to deliver fieldwork within six hours, and it’s a very large audience in numerous countries, having quick access to research-ready respondents is a must, because you can’t go back to the days of manually setting it up – it needs to be done in a programmatic way,” says Andrew Cayton, global chief operating officer at Kantar’s Profiles division.

The division’s online panels support the majority of its fieldwork; however, global scale and speed has become more important over the past few years, says Cayton, so the company created an integrated private network, including some invitation-based panels, as well as those that are more programmatic in nature. It also uses other platforms to access sample, depending on the needs of the client and project.

“Depending on the research brief and type of work, we have our own panel to tap into, but we also have the ability to programmatically buy the audiences at scale through the private network and other platforms,” explains Cayton.

SurveyMonkey’s aim is to offer the same level of insight that you could expect from a market research agency. The key difference? “We want to do it a lot faster,” says Taleen Connor, senior director of business operations. To do that, it needs to use programmatic sample. “The data collection is really critical because, if there’s a delay in getting the data, that can really slow down the whole process, and we think it’s a really big differentiator to get those insights more quickly so you can make decisions faster.”

The business has its own proprietary panel and sample, but a large part of its approach to sample is working alongside sample partners to help it reach its desired scale “more quickly”, adds Connor.

Perception questions

There are three levels of understanding and uptake of insights automation within enterprise organisations, according to Kokoszka. The first group believes that research is becoming less relevant in the boardroom because of the availability of other, quicker data signals, including those from social platforms. For those clients, he says, it’s nothing to do with cost or poor quality, it’s just that research is too slow.

“That category is very keen on digital transformation of insight and, therefore, the need to apply technology and break away from the old ways of doing things,” says Kokoszka. “But you need to be brave, in large enterprises – nobody gets bonuses for getting it right, but lots of people get fired for getting it wrong.”

There is a middle ground of organisations that understand the benefits but are more hesitant to act; they will eventually move towards automation, but will take longer to do so.

Then there is a large third group of companies that is content with their current status quo, says Kokoszka. “It’s kind of comfortable not to automate and you can make up a lot of reasons about quality, about the need to be personal – or, as I had somebody tell me, ‘the maths is too difficult for a computer’.”

While the average tenure of the chief marketing officer at large organisations is around 18 months, heads of insight remain for longer. Kokoszka argues that there is subsequently less pressure to transform, so automation is not generally viewed as integral to the brand or the survival of the insights function. Rather than a system change, it is seen as a tool, such as email.

Dunbar notes that industries that have previously undergone large transformation – for example, advertising and finance – have experienced the same challenges. Rather than focusing on programmatic sample as a tool, for example, the focus should be on the whole system, he argues.

“The biggest learning curve is getting people to think about technology, system architecture and process re-imagination, instead of thinking about sample feasibility and costs. We still spend too much time talking about what sample we have in our open exchange – it is obviously important, but I think the more interesting conversation is: what does the right system architecture look like? What does the right tech stack look like, and what does an ideal operating procedure look like to fundamentally transform the way you are collecting your data? That is a much more valuable conversation to be having.”

DIY research has changed clients’ knowledge of the process, so the need for transparency is high, says Natalie Smith, global head of delivery at Hall & Partners. “That has put a lot of pressure on the industry, which has changed how we think about automation,” she adds.

“Clients want two things – quality and speed, and they don’t want to choose, and I think that’s fair. DIY has been an accelerant; the pandemic has been an accelerant for automation. It puts pressure on the industry to come up with more innovative solutions.”

Quality procedures

While there’s a clear imperative for insight to be delivered faster, it still needs to be robust. Ensuring that respondents have given their consent, that sample is compliant with privacy legislation, and tackling the issue of survey fraud – where click farms and bots generate fake respondents – are all vital concerns for online researchers.

Ipsos Digital has developed an integrated DIY research flow, offers assisted DIY, and connects to its sample vendors through application programming interfaces (APIs).

Can programmatic sample address survey fraud? For Andrei Postoaca, chief executive of Ipsos Digital, the programmatic process does not have much to do with quality procedures per se. “Fraud is everywhere, it’s just a matter of accepting it and making sure that you put in a lot of procedures to diminish it and, hopefully, take it out completely,” he says.

Postoaca likens survey fraud defence mechanisms to the Swiss cheese concept, where an organisation’s defences against failure are represented as slices of cheese. “If you slice it, you will have holes in it, but if you put 10 slices one after the other, you will most likely not have one hole going through the whole flow. That’s how I see fraudulent respondents.

“Each and every method and tool you apply will have holes, but put 10 methods and tools on top of each other, and they will catch some respondents that will be fraudulent.

“Rely on just one slice, and you will certainly have holes.”

Postoaca adds: “I was speaking with a client this week and the whole discussion was around the trade-off between narrow samples – reaching the needle-in-the-haystack kind of samples versus really having solid quality. What really worries me is how to put in a sufficient amount of validations.

“At the end of the day, market research is worthless unless it’s really good quality. If I’m not comfortable with the quality, I might as well guess myself.”

A helping hand

Lucid is seeing more clients coming directly to the company to manage their sample themselves – for example, by integrating its platform into their own website.

“We have seen more people taking hold of things and wanting to take ownership of the research process,” says Simon Beedell, executive director, EMEA, at Lucid. Traditionally, clients would work with Lucid on a managed-services basis and then move to a more direct approach once they have become familiar with the platform. But, explains Beedell, “what we’ve seen more of in the past six months is people coming directly to us asking to skip that step”.

That’s not to say everyone wants to be completely hands-on. “Some clients very much prefer managed services,” says Beedell. “They prefer to have the right experts on the case in terms of the fieldwork and to have that reassurance that someone is on the other end of the phone or email, ensuring things stay on track. That human element is still quite important.”

SurveyMonkey sees a lot of clients moving to a DIY approach once they have become acquainted with the platform. “It’s so much quicker for them to get insights, especially in verticals where it really matters, such as financial services, where they’re making investments and it’s a matter of hours before they need to make the decision,” says Connor.

It’s clear that there is no one-size-fits-all approach when it comes to DIY research. Beedell compares it to visiting a travel agent on the high street or booking a holiday online, directly with an airline: “Just because I don’t understand the travel agent benefit, it’s not to say a travel agent is totally irrelevant.”

Ipsos Digital’s Postoaca says the company wants to ensure that clients can do as much – or as little – themselves as they please for a particular project. “I might be a total DIY fan, but today I have no time because I’m in a meeting and I need to launch those five questions, and I want to just send an email to someone. DIY is not black or white; it’s not yes or no. It’s what the client wants for that project.”

Sample automation isn’t right for every piece of research. Projects that are more straightforward in nature or have higher incidence rates are better suited to programmatic sampling.

“If you’re an agency or a client, and you want a quick, nationally representative copy test, programmatic sampling is just awesome,” says Hall & Partners’ Smith, adding: “It has limitations. If you’re looking for quite niche audiences, you need to augment or target in a different way.”

As the technology matures, more complex use cases could become available, but for now, the balance between complexity and automation remains a bit of a trade-off, says Kantar’s Cayton. “The technology and the ability to automate are there, but the challenge is making sure the client doesn’t overcomplicate the research design or question set.”

He predicts this will change in the coming years, in terms of what can be achieved. “But, today, if you design something that’s a very long survey – that’s also a very complex sample design and a niche audience – it’s going to be very difficult to automate, because the audience will be difficult to get programmatically,” Cayton adds.

“If the survey is long, even if you get the audience, the respondents won’t necessarily finish the survey – we’ve seen a real shift towards respondents responding only to very short, 10-15 minute surveys.”

When you take into consideration recent research from Dynata, Kantar, Toluna, Lucid and the MRS highlighting the importance of survey design optimised for mobile – as well as increased competition for respondents’ attention – this could point towards a trend of online surveys becoming shorter and simpler.

Future-proofing

The pressures of Covid-19 have already accelerated the process towards industry automation, as the ability to access audience insight at scale and make recommendations quickly for clients becomes more necessary in times of change.

Dunbar talks about the “paradox” of brands seeking to become more personal while still understanding audiences at a bigger scale – a task made more challenging with third-party cookies being phased out, making it harder to identify individuals.

“Market research is, at times, becoming more about personal research. Those two things are juxtaposed – operating at a bigger scale and trying to get personal,” says Dunbar.

Because of this, he argues, there is a need for the industry to work together. “I believe the more the industry comes together to partner on all of these kind of things, we will solve our end customers’ problems faster than if we try to do things in isolation.”

Cayton says: “The lifespan of data is getting shorter. Clients are in tough spots because of the pandemic and they’re really trying to understand how people’s buying behaviour has changed for good – and if it’s changed, how it’s changed. The only way to do that cost-effectively is to automate.”

He predicts an increase in both self-serve and partnership models for the buying of sample.

“I think we’ll see more offers on the market in the next 12 months,” says Cayton, “either offering DIY access or what we think will win out, which is ‘do it together’ – working with the clients and designing something bespoke for them.”

This article was first published in the JULY 2021 ISSUE OF IMPACT.