The challenge of relevance

x Sponsored content on Research Live and in Impact magazine is editorially independent.
Find out more about advertising and sponsorship.

“The people who drive our cars have dogs, they have kids and they have gloveboxes full of tissues, wet wipes and cables,” says Steve Hill, consumer insight manager at Jaguar Land Rover (JLR). “The more data you get into the business, the harder it gets for people to see customers for the humans that they are. You see people as being numbers on a spreadsheet, or points on a data chart.”

The fear that insight could become an impersonal, data-driven function was the inspiration behind a project for which JLR and its agency, C Space, recruited a panel of 100 drivers. These motorists were asked to interact, give feedback, join video calls, and even go for a pie and a pint – but, significantly, not with the insight team. JLR’s panel communicates directly with the designers and engineers, sharing their happiness, frustration and any other emotion, often without mediation through research professionals.

“If we can use market research to start giving people a bit more of a visceral understanding of how our customers think, and how a relevant person thinks, then it’s a win,” Hill says. “Even when they’re telling you the bad stuff, they’re doing that because they want you to improve.”

Creating small-scale intimacy is one solution, but this does not meet the challenge of recruiting and incentivising larger panels. At the other end of the scale, Research Now (RN) is meeting the challenge of relevance by recruiting panels differently. To join RN’s panel, you have to be invited by a brand partner; in the UK, for example, that means Nectar. “One of the primary aspects of driving quality is our recruitment model,” says Ryan Jantz, senior vice-president, panels and partnerships, at RN.

Personal, iterative, focusing on bitesize insights delivered, via technology, direct to the people who use them, the work of Hill and C Space is just one example of how the best insight teams are stepping up to the challenge of keeping their work relevant to both the client and the customer. On the client side, research conducted by Professor Niels Schillewaert and his colleagues at the Vlerick Business School suggests that only half of research projects lead to change, and that roughly the same proportion of researchers think that their work affects decisions. On the customer side, the more we know about how decisions are made – and how opinions are reported – the more some methods of collecting data seem like a battle against the habits and instincts of ordinary people, rather than a reflection of them.

As long as new methods of gaining and using insight have been available, this has been a challenge that the industry has met. But as the datafication of customer relationships has unleashed new sources of insight, often generated direct from the clients’ internal systems in near real time, the role of externally generated insight needs to keep up.

Adjacent insight providers – notably those selling big data as a solution to how to make faster, better decisions – point out that traditional research doesn’t match well with how companies prefer to act, and sometimes isn’t finding out the most useful information. For example, in its report Big data: The next frontier for innovation, competition and productivity, McKinsey contrasts the broad segmentation of market research with the immediate, actionable data that clients such as department store Neiman Marcus are using to reach customers. “Retailers can now track and leverage data on the behaviour of individual customers – including clickstream data from the web. Retailers can update this increasingly granular data in near real time to adjust to customer changes,” it says.

If the insight function has to give a seat at the table to data scientists, arguably, this is a good thing; if all data is seen as relevant to decision-making, then research and big data have a common fight against management gut feel. But research agencies are also in a struggle for scarce talent against technology companies that suddenly seem very relevant as a source of methods to make better decisions. If they cannot demonstrate they are operationally and strategically important, they are at a recruitment disadvantage too.

Perhaps the next evolution of the insight business isn’t so much about finding new techniques, but embedding existing methodology more relevantly in the experience of those who give information, and those who could benefit from using it.

Who makes decisions?

In 2010, Bain & Company surveyed 760 companies, most with revenues greater than $1bn. It was surprised to find that most company reorganisations had no benefit, and most destroyed value. Its survey showed that the ones that prioritised the view of the firm as a decision-making organisation had far better outcomes. So Bain & Company began to recommend to its clients that, if they were planning to restructure, they organise around decisions, rather than hierarchies or functions.

Not every company follows this advice, but some are taking it seriously. For those that do, plugging insight into the decision-making process will be essential.

“On an annual basis globally, our industry invests $43bn on generating insights, but only a very small piece of that money is invested in activating those insights,” says Tom De Ruyck, managing partner at InSites Consulting. “And it’s a bit like in advertising; we know that half of the money we are spending will not lead to action, we just don’t know, at the beginning, which half that will be.

“More clients feel the need to change the way they do things because they see that the old ways aren’t working any more, or aren’t as effective as they used to be. They are in the situation where the external world is more complex and faster-moving than ever before, but their internal world is still moving slowly.” De Ruyck has put this into practice through his relationship with Ikea, which now spans five years. Ikea had this feeling, he says, and, as a result, decided to reorganise the way it handled insight, to bring it closer to decision-makers without the internal insight department acting as a gatekeeper. “Then you need to do two things. You need to look at how you generate insights – what tools we have, what research programmes are in place, and how we need to change these to be more open and agile in how we catch consumer insights. Then you need to look at how to activate insights in the business. How can you involve more people and have more impact on the business?”

Part one meant streamlining the numbers that Ikea tracked to three measures of business performance. The performance in those numbers could inspire further investigation, but emphasising co-creation rather than directed surveys. “It’s also about allowing consumers to bring topics to the table themselves,” De Ruyck says. “It’s about what you want to discuss, as an organisation, but also about what the consumer wants to discuss.”

This process is only one aspect of how insight became more relevant at Ikea. Alongside this reorganisation, the company decided to share its insights more directly with more people. It reasoned that, in a retailer, far more people are making decisions every day – for example, how to speak to customers – so they could benefit from knowing what customers need.

“We are democratising the use of insights, because we are making it easier for people to find them, to understand them, and to absorb them on the one hand, but we are giving access to more people – or giving people more access to those insights as well, which I think is needed,” De Ruyck says.

“If you want an organisation to move faster, and to make better decisions faster, you need to give more autonomy and empowerment to employees, but you also need to give them the necessary inspiration and insights.”

Few employees want to read charts and reports, however. So Ikea and InSites Consulting concentrated on creating ‘memes’. These are simple insights based on the data that would be meaningful at every level of the organisation, and which attract comment from a wider group of employees, who contribute ideas on how to improve based on the memes.

InSites has measured the uptake of its research findings, using these methods internally. The result of spending more time on how to communicate its results – and who to communicate them to – is that 85% of its findings are acted on. The key, InSites argues, is to look beyond the usual community of people who use research and find out which other decision-makers would benefit from knowing about it, and how they use information in their jobs.

Rather like the successful decision-makers in Bain’s sample, this partly depends on the client being able to structure its business around decision-taking, and let insight flow to where it is needed. “If they want to move faster – which they have to, because the external world is moving faster – they need to have less hierarchy. They have to give people more autonomy, more empowerment,” De Ruyck says.

Delivering the message

“Many market research companies have manoeuvred themselves into an almost impossible position by not changing. They have a lot of pressure on their work process,” says Jeroen Rietberg, co-founder of Intellex Dynamic Reporting and CFM International, which supplies software to the market research industry and end clients to automate reporting, create dashboards and combine research sources or voice-of-the-customer tracking.

Rietberg identifies three dimensions of research that are important to the client: quality, efficiency and effectiveness. The first can be narrowly defined as the methodology used, and representativeness of the analysis. The second, responding to market trends, as being able to get results quickly and at lower cost. But the third – effectiveness – has not been given the attention it deserves, Rietberg argues.

“Effectiveness means what is actually done with what you deliver. How much impact do you have, and how do you know? Do you ask your clients what they do with your data, and how much effort they put into translating what you give them into something usable and actionable? That is a big problem,” says Rietberg.

One of the most important measures of the effectiveness of market research is its constant presence when decisions are being considered. Hill, at JLR, jokes that, after receiving a brief from an area of the business, a typical insight department “spends three months living and breathing the research, but as far as your internal clients know, you’ve gone quiet and just buggered off”.

Rietberg argues that too little focus on how those internal clients want information has resulted in technology companies moving into the market. “The types of companies who are now delivering the voice of customer are often technology-based companies, who might have been supplying technology to market research, but who were definitely not market researchers themselves. They have just taken over that whole area of customer satisfaction.

“The reason they could do that was because they thought about the way in which information has to be presented to the client, to make it usable for them, workable, and immediately applicable to the organisation. That’s as simple as it is. If you want to make an impact with anything that you do, it has to be used. If it’s not used, it can never have an impact.”

BrainJuicer has never been an average agency, but it has always marketed its propositions on the basis that – if you want to find out what people are really going to do in the future – asking them is often not a good idea. Instead, data science discovers in greater detail what they are doing.

Questioning the methodology

As a contrarian, BrainJuicer liked to quote Daniel Kahneman before it was cool to do so. It’s still a tough sell – building a business by telling clients they are measuring the wrong things – but, says chief operating officer, Alex Batchelor, it’s the responsibility of an agency to challenge the beliefs of the client if research is to remain relevant.

He quotes Kahneman: “Our comforting conviction that the world makes sense rests on a secure foundation: our almost unlimited ability to ignore our own ignorance.” Or, in Batchelor’s words: “It’s like when mums can’t believe their children are murderers.”

When research makes the news, it’s usually about polling, so the news hasn’t been good in the past nine months. We can argue that polling is a small part – underfunded and unrepresentative – of the business, but Batchelor points out that there’s something in the criticism. The industry is still asking people what they intend to do, relying on a rational, considered response at the point the research is done, while, all the time, we discover more about the limitations of this technique.

This is a fundamental challenge to the relevance of research, Batchelor argues. “Purchase intention is kind of a dirty secret: we all ask it, even though we know it doesn’t correlate strongly with anything. We should all be looking at something that is more predictive. We believe the tide is turning in our direction. Many agencies know that emotion matters; they accept it, but stick questions about it down at number 51 or 52. If you’re taking the survey, by the time you have got there, you’ve lost the power of feeling.”

How to sell it

Go to ZappiStore’s website and the first thing you see is: ‘Traditional market research takes too long and is too expensive.’ Who says? “Everyone I talk to,” says Christophe Ovaere, the chief marketing officer at ZappiStore, “They say, we can’t justify that kind of spend and that lead time any more.”

ZappiStore is an automated platform on which agencies can offer tools to deliver bitesize pieces of research quickly – on demand – to clients. It’s neutral about which methods or providers host their tools. At the moment, Millward Brown, Pointlogic and MMR are there. Ovaere talks about how some very small agencies are developing specialist, niche tools to sit alongside them.

But it is not simply an aggregator; ZappiStore was created in response to demands from clients – which include Coca-Cola, LG and Ford – to work differently with their research providers. “We will deliver shorter, snappier and faster work,” Ovaere says. “Clients can use it cyclically and iteratively, which means they are agile.”

He tells the story of how a client wanted ad testing on a Friday night. “She said she had four ads, and ordered the tests as she was leaving the office. By the time she reached the bar, the reports were ready. She could discuss the results on Monday, change the ads, and test again: it makes test and learn possible.”

Ovaere disagrees that this makes research more disposable. It merely fits with the way the clients treat the flow of information, he says. “Agencies innovate a lot, but they innovate around methodology. What are you going to change that is so groundbreaking that the client falls off their chair? Not much. But there has never been innovation around process, cost and time. We are one of the last industries that hasn’t optimised the delivery process. In some respects, we’re still running a manual sweatshop.”

Another client has just set up a ‘data war room’. “The market research agencies should be running that room, but how many are in it?” Ovaere asks. “None.”

BrainJuicer has located one of the tools that ZappiStore is marketing. Batchelor likes the idea of providing behavioural insight rapidly, at low cost. “Let’s see if we can sell it to people we couldn’t afford to sell it to otherwise,” he says.

“When we started, we dealt with clients who didn’t want desktop self-service; they wanted to phone people and ask them to find out things. But SurveyMonkey has dramatically expanded the survey market to clients that – when they phoned up the big agencies – basically got laughed at.”

Even among large clients, Batchelor has seen a change in tastes. “I suspect there are some things for which the client wants to phone up brainy people and ask them to find out something, and there are a lot of other things that are like a sort of monitor. It goes on and on, and suddenly goes ‘ping’ when you’re out of an acceptable level of performance – then you investigate.”

Who’s taking your surveys?

‘Has the US become over-surveyed?’ asks the headline on an article in which Irving Crespi, of Gallup, says: “I wonder whether we are reaching the point where we are saturating the public. It is a very serious problem that threatens the long-term viability of the survey profession.”

The article points out that over-surveying leads to resistance, especially from the young and old, the well-educated, the very rich and the very poor, who are all under-represented in samples. This bias – or unwillingness to engage – may skew polls for the upcoming election, it warns. The American Statistical Association laments that most surveys are trivial, and that they have become “an exploitative exercise” for the public.

That’s from the 29 October edition of the New York Times – October 1975, that is.

It’s also now more than a decade since comScore’s research into online panels found that 30% of the surveys in the US were taken by 0.25% of the population, and that those 0.25% belonged to, on average, seven panels each.

Yet, of course, we’re still hearing the problem that there are too many surveys, asking too many questions, with too little validation that stated preferences are genuine or representative. Surveys have become an unwelcome accompaniment to any online activity – an annoying buzz that tells us our opinions matter, without sounding at all convincing.

Recently, there has been much worry about the survey-taking behaviour of millennials; in the February 2016 edition of Quirk’s, for example, Dan Coates, MaryLeigh Bliss and Xavier Vivar, of Ypulse, analysed the responses from their panel of 18- to 34-year-olds, and found their attention span for surveys shows a sudden drop after around 15 minutes.

In the past two years, though, this has dropped alarmingly. “Reviewing our recent syndicated surveys shows that population has been dropping off closer to the 13-minute mark,” the company says. Half the respondents now complete surveys on a smartphone, but they tend to drop off earlier – Ypulse recommends surveys of 8-10 minutes.

Recent polling problems highlight the flaws of the data we collect. On the one hand, it lessens confidence in surveys among decision-makers, although part of that stems from a lack of engagement from the public. The relevance panic that was gathering steam in the mid-1970s hasn’t caused the end of research but, equally, the industry hasn’t solved the problem with panels.

At RN, Jantz says that his invitation-only recruitment model is intended to reduce this problem. “Nobody is getting rich taking surveys, but the incentive is important, and that’s why we focus on loyalty programmes, or brands with strong incentives or rewards. The consumer joins because they value the incentive that the partner offers. They’re engaged with that brand. They value that currency and they are willing to share their time to earn that currency,” he explains.

Focusing on loyalty brands for recruitment means the incentive (survey-takers earn points for vouchers) is valued, but RN stops keen survey-takers from participating too often – or from taking multiple surveys at a sitting – because each type of behaviour implies a reduction in engagement. It’s a tricky balance to strike, Jantz says.

RN is also working to increase the relevance of its surveys by developing technology so respondents are not constantly asked the same boring, but important, questions – for example, qualifying questions, or details of status that could easily (and more accurately) be sourced elsewhere. Asking only what you need keeps surveys short enough for millennials and avoids the triviality that probably still enrages the American Statistical Association.

Finally, targeting a pre-engaged population is a way to source scarce panel types; as in 1975, some segments are hard to engage. “This allows us to get segments of the market that no-one else can get at,” says Jantz. “Business-to-business research is a very challenging thing to do because, if you’re recruiting out in these open-recruitment sources, the IT decision-maker tends not to be hanging around those places. We’re uniquely able to get at those people. That’s how we craft our partnership strategy. We look for under-served or under-represented segments of the population.”

The result, claims Jantz, is that “an order of magnitude” fewer respondents are excluded for quality reasons in comparison with other panels. That means better relevance for clients, but also suggests that the panellists are more engaged. Response quality, Jantz argues, is a better driver of value in the long run than sweating the firm’s assets, when the asset in question is a panel of people who you’re trying to engage. “The optimal management of an asset [traditionally] is to get as much out of it as you can. We recognise that’s not the best way to generate a high-quality research asset.”

That level of engagement is what Jeremy King, the CEO and founder of start-up research provider Attest, is seeking. A trained research geneticist, who was previously a consultant at McKinsey, King says: “I saw the same problems with how insights are used in every organisation I came across.”

Attest is building a semi-automated platform to help solve these problems, starting with panel recruitment. King wants to bring more of his science background to bear on the recruitment and retention process so that, in hard-to-find segments, there are representative samples to survey. He is also building a platform so that non-specialists can create credible surveys, designed so that panellists only get involved in research that is relevant.

Pre-launch, King is cagey about the rewards for participation, but he points out that high-value participants – especially wealthy ones – aren’t motivated to engage so much by cash or voucher incentives, as by the desire to see what has been done with their input – to have “a seat at the table”.

The tools that inform scientific research – longitudinal surveys taking a minimum of time, with quick feedback, presented as a regular input to decision-making – will create an emotional connection between panel and client. “We’ve seen a lot of agencies building the faster horse,” says King, “We’re trying to build the car.”

Whatever that car eventually looks like, King is part of an emerging movement to create relevance in research: prioritising nuggets of insight, delivered where and when they are needed in the business; using technology platforms to automate the reporting process; and prioritising smaller-scale intimacy rather than large-sample incentivisation. Research is fighting back against those who question its relevance in a world of behavioural and operational data.

For the decision-maker, the respondent, the future researcher, or simply the curious customer, there’s no easy solution to making insight relevant, because relevance is different for all of them. They know it when they see it, which is why the process of defining a project too tightly, or restricting that process to an internal insight function, is often risky.

For everyone dedicated to ensuring relevance, there’s one common factor: research seems to be most relevant when the right people at each end can communicate without barriers – a client-agency partnership structured around decision-making, not hierarchies of reporting.

“That ability to lose control is really, really important,” says Hill, at Jaguar Land Rover. “When you do that, nine times out of 10 it tends to work. I’m an absolute control freak. It’s quite hard for me even to say this, but you’ve got to be prepared to let go, and hope that it comes out in the way that you’re expecting it to.”