Mastering the wave: How to future-proof the insight sector

x Sponsored content on Research Live and in Impact magazine is editorially independent.
Find out more about advertising and sponsorship.

For Clare Michelmore, UK insights lead at FMCG giant Kimberly-Clark, it all started when someone showed her how ChatGPT could be used to write silly poems. Since then, she’s gone from limericks to complex research applications, using generative artificial intelligence (AI) tools to transform the way she works – and she’s excited about the future potential of such technology.

“Insights is one of the best industries [for these tools] because there are so many good, interesting applications,” she enthuses, name-checking techniques such as eye tracking (for which AI has a close correlation with humans, she notes), qualitative probing (where generative AI follows up on open-ended questions with research participants, generating in-depth answers) and assistance with question writing (taking away some of the mental load) as particularly useful in her experimentation journey so far.

As Michelmore’s story demonstrates, generative AI is already having a profound impact on how many insight professionals are working. The sector is ripe for disruption, with the potential of these tools to speed up data analysis, automate insights generation, produce compelling data visualisations, and even recommend campaigns based on its insights. No profession or industry is immune, however. According to a report from Bloomberg Intelligence, the generative AI sector could grow to $1.3tn over the next 10 years, from a market size of $40bn in 2022. ‘AI’, the abbreviation of artificial intelligence, was named the 2023 word of the year by Collins English Dictionary, while Cambridge Dictionary chose ‘hallucinate’ and updated its definition to include AI programs providing (or, rather, making up) misleading or incorrect information.

In the research sector, much discussion has centred on the potential of these tools to transform the work itself. Less has been said about the skills required to future-proof the industry and the people working within it at a time of growing reliance on technology. With the possibilities that technology offers advancing at an intense, almost breathless, rate, how can researchers ensure that the future is one of humans and machines working in tandem, rather than machines going it alone?

First, though, it’s worth re-establishing what generative AI does well – infinitely faster, if not better, than humans – and how researchers can benefit. Ben Davis, head of data, AI and research tech at Opinium, explains that AI can help tackle complex research problems, such as how to extract meaningful insights from large volumes of unstructured data, at speed and scale.

“Insights workers will be empowered to perform technically complex tasks, such as advanced statistical analysis and iterative design of open-text code frames,” he says. Research from Harvard Business School and Boston Consulting Group in 2023 found that consultants using AI produced 40% higher quality results, and completed tasks 25% faster, compared with a control group. “AI tools not only help us automate tasks and complete work faster,” adds Davis, “but they also help us deliver higher-quality study design and research analysis to our stakeholders and clients.”

Phil Sutcliffe, managing partner at Nexxt Intelligence, describes the technology as having game-changing potential compared with tools that have come before. “There’s been a mantra for the past 10 years about using technology to be faster, cheaper, better – but mostly it’s been faster and cheaper, sometimes with detriment to quality,” he says. “But generative AI allows us to understand people better at scale.” This, he adds, frees up researchers to focus on more impactful – some would say rewarding – work, such as designing research and analysing and communicating it in a way that adds value and leads to better business decisions.

“This disruption is fundamentally a good thing for the research business, because it forces us to connect with what we are great at,” agrees Zoe Fenn, leadership consultant and founder of You Burn Bright. “Those strengths might have been underused for a long time – we used to spend so much time churning through data to get to what we love to do. This shift will allow those brilliant minds to get back to what they are great at, but haven’t had much time to put their energies into.”

The benefits of researchers engaging with this technology is clear, and the research sector is well placed to take a lead on embracing new tools. Betsy Fitzgibbons, shopper insights lead, e-commerce and new transactions at Mars Wrigley, believes the insights function is the one “best suited to pilot, test, kick the tyres and embrace AI technology for the good of the business, identifying where it can add strength, power and effectiveness”. But, she adds, it is incumbent on insights teams to “recognise and call out AI’s limitations and pitfalls, as well as build guidelines”. (More on that later.)

So, what are the specific ‘technical’ skills insights professionals need to engage meaningfully with this technology? After all, despite their impact and speed of development, many of these tools are in their infancy.

According to a 2023 survey by Microsoft, industry leaders believe it is essential that employees from all professions and sectors learn when to leverage AI, and how to write great prompts, evaluate creative work and check for bias. Fitzgibbons flags data analytics as an essential skill for insights professionals and one that won’t be disappearing, even as machine capability increases. Opinium’s Davis highlights a culture of continuous learning, with an emphasis on data and digital literacy.

As Microsoft’s survey found, prompt engineering – the process of structuring text instructions so they can be understood and interpreted effectively by a generative AI model – is increasingly important. “To be most useful, generative AI needs clear objectives and context,” explains Harley McKee, business intelligence manager at Opinium. “Better outputs and ideas come from better reasoning and thoughtful instructions. Similar to how the best managers inspire great work by defining roles, removing ambiguity and setting clear goals, researchers must communicate with large language models (LLMs) as if they were a person to whom you were delegating important work.” When it comes to learning how to do this effectively, he recommends “getting your hands dirty”, jumping in and experimenting with the technology.

This is something Michelmore has experienced. “The first time I briefed a bot to do an end-to-end piece of research, I realised it hadn’t understood my brief when I got the research back,” she recalls. “If I’d been briefing a human, they would have asked questions. But my business questions remained open – which was fine, as it was a pilot, so I was in a safe space to learn. It was a great tool; what messed it up was my inexperience.”

Michelmore advocates piloting and creating safe spaces to learn and fail. A portion of her budget is ring-fenced for doing just that, and the team has quarterly development days to focus on learning new skills. “The most important factor is the willingness to try something new,” she says. Curiosity about the possibilities of new tools and an appetite for experimentation – with the necessary guardrails in place – is a critical mindset for researchers to adopt.

Steering the ship

For those who are less technically minded, Sutcliffe has a reassuring message. He believes insights professionals, on the whole, do not need to understand what is inside the ‘black box’: it’s about knowing how to leverage the technology and mitigate risks. “AI will increasingly become about buying applications,” he says. “Not everyone is going to need to be an expert in how AI works; you are going to need to be expert in how you can use it to your advantage.”

While there is a benefit to understanding prompt engineering, Sutcliffe doesn’t think it is essential for all researchers, given that solutions will increasingly have the AI built in. “The job of a researcher is how to address the client problem, find the insight and communicate that effectively,” he explains.

In other words, it is more important to know how to leverage some of those uniquely human (for now, anyway) skills. For Sutcliffe, the most essential are what he calls the ‘four Cs’: connection, critical thinking, creativity and communication. Connection is about empathy – the ability to connect with people on a human level, something machines cannot yet achieve. Insights professionals need to “add a human layer to data to bring it to life”, says Fitzgibbons, “as well as be the human conduit to maintain that empathy component that is necessary… to connect with consumers”. They must also be “the voice and champion of the different voices of the consumers, [ensuring] businesses don’t just go after numbers”, she adds.

Critical thinking is perhaps the most essential skill when it comes to mitigating risks associated with AI. Assuming tools are infallible is not just ill-advised – it’s dangerous. There are plenty of examples of generative AI ‘hallucinating’ inaccurate or misleading information that could be damaging to consumers and organisations. It frequently hallucinates false legal information, for instance, and AI-generated imagery has been found to be biased, amplifying gender, racial and other stereotypes.

Insights professionals must be aware of such risks and ensure that a human is always in the loop to watch out for errors and inappropriate suggestions. “Automation does not relieve us from validation,” says Nick Rich, until recently vice-president, insights and analytics at Carlsberg Group. “We have an obligation to ensure [AI-generated outputs] represent a true perspective of consumer behaviours, needs and desires.”

The issue to recognise, says Shari Aaron, executive vice-president of growth and innovation at Radius, which has worked with Mars Wrigley on AI-based projects, is that we cannot keep tabs on the millions, even billions, of data points from which AI is drawing. “Given the scope and large-scale abilities of the LLMs to take in data, we have to be uber-cautious about the validity of the information included before we trust the outputs from these models,” she warns. “There is no way to know the accuracy… LLMs are so opaque.” Anderson offers a note of caution around inputting proprietary or sensitive information into open-access tools such as Bard or ChatGPT: “Security of confidential information is our first priority.”

McKee agrees on the need for caution, advising: “Researchers must be careful not to ‘fall asleep at the wheel’, letting the AIs do everything for them or copying outputs without critically reviewing them. We need to grow as strategic thinkers. While AI tools can help us run our metaphorical ships, we always want to make sure we’re the ones who make the final decision about where to steer.”

While generative AI can produce ‘creative’ work (see image generators such as DALL-E), true creativity remains in the realm of things humans can do better. The consensus is that using AI frees up more time and brainpower for humans to focus on creative and strategic tasks. But that doesn’t mean AI can’t provide a useful starting point for creativity, whether by analysing reams of insight to suggest concept angles and taglines, or suggesting survey questions that a tired and overworked researcher may have missed.

“Generative AI tools help jumpstart creativity and offer a sounding board for our insights consultants to sharpen their research reasoning, which, they say, makes tackling the most complex research challenges feel more manageable – and seriously fun, too,” says James Endersby, chief executive at Opinium and chair of MRS.

Once you have developed ideas and analysis, communication remains, for many, the final hurdle. As Sutcliffe says: “Having insight is one thing, but persuading organisations to take action on those insights is another: it requires great storytelling and communication skills.” These are not a given, and communication is an area in which many commentators feel insights professionals must up their game.

“We still have a long way to go to develop those skills,” says Fenn, about the importance of communicating effectively with colleagues, stakeholders and clients. “We’ve been working in silos. You don’t come to an insight on your own.”

That enhances the importance of what Fenn refers to as social capital: the connections between people. To briefly skirt another hot debate, the increasing use of AI could mean there is more value in spending time together in a workspace, especially for early-career professionals who benefit from mentoring (formal or informal) and observing and collaborating with more experienced colleagues.

Remembering strong foundations

Communication, creativity, empathy, critical thinking… these are all transferable essential skills. They are sometimes described as soft skills (rather unfairly, as many leaders know they can be some of the hardest to demonstrate), but they are the skills a machine will find most challenging to replicate. It would still be a mistake, however, for insights professionals and employers to neglect the foundation of market research.

When musing on in-demand skills, Sinead Hasson, chief executive at specialist recruitment agency Hasson Associates, points out that companies will always need classic research skills. “The ability to be a great qualitative or quantitative researcher will always be valuable,” she says, noting as equally useful those skills around open source and social media data analysis, people, communication and leadership, and business nous.

Rich also highlights business acumen, pointing out that client-side leaders in particular need to approach AI from a ‘business problem first’ perspective rather than getting over-excited about the technology itself. “Most important is to seek to define and solve the business problems identified with AI in a methodical way,” he advises. There is also a need to remain strategic when it comes to investment. “I’ve seen a lot of businesses talking about AI this, AI that – and what we’re doing is potentially making the same mistakes we made with big data,” says Rich. “It’s really important to understand what the business problems are that you have, and then work out which AI solutions offer you a better, faster, maybe cheaper way to solve that problem.”

Michelmore says: “Don’t stop learning traditional research skills. AI is like swimming; I can swim faster with flippers, but the flippers can’t swim without me. AI means we’re going to be able to do more things faster, but I don’t think that you can take humans out of it anytime soon.”
With the risks of relying too unquestionably on the outputs of AI, researchers must have a solid knowledge base to be able to question and evaluate outcomes. Deciding on what skills to develop is a challenge for leaders in the sector. Soft skills are transferable and should be invested in, as these will help people work cross-functionally and are not going to be replaced by machines. However, people with those skills might not have gone into market research traditionally, instead choosing marketing or consultancy-type roles. As Sutcliffe points out: “The sector is going to have to be more focused on recruiting intelligent, capable, curious people and helping them develop those skills.”

A final rule to live by when it comes to working in tandem with machines, says Preriit Souda, of PSA Consultants, (see boxout, below0 ), is not to try to compete with the machine. “Machines come and go; it’s the ability to think that matters,” he says. “Use these tools to enhance your ability, not compete with what they do faster and better.”

A 2023 IBM report found that leaders expect 40% of their workforce will have to retrain or reskill in this new world, and included a powerful quote: “AI will not replace people… But people who use AI will replace people who don’t.” Asked for his advice to researchers looking to future-proof their careers, McKee says: “Be curious and never stop learning.”

It’s not about letting technology take over, but considering how you can use it to do your job better. Don’t fear the machines, but don’t trust them implicitly and uncritically either. Future-proofing is about making sure you are one of the people who is aware of and curious about how to work in partnership with them.