Virtual engagement: Can AI improve surveys for participants?

x Sponsored content on Research Live and in Impact magazine is editorially independent.
Find out more about advertising and sponsorship.

The metaverse. ChatGPT. OpenAI. Chatbots. Apple Glasses. Google Bard. Much has been written in recent months about the potential transformative impact of artificial intelligence (AI) and augmented reality (AR) on the market research industry and wider society. It has become increasingly hard to separate the wheat from the chaff when it comes to opinions on the subject, with little evidence as to what the likely role either technology will play in our daily lives and in the workplace.

Heineken wanted to see what impact technologies such as AI had on engagement with consumers. To try to address the evidence gap, MMR Research and Heineken set up an experiment to see how AI technologies affected market research surveys and whether they could enhance the reach of market research, especially with a younger audience. The study followed an MMR Research-funded experiment on AR’s impact on survey engagement.

With much of the industry’s experimentation in AI and AR currently focusing on qualitative research, the researchers chose to conduct the study using quantitative methods. The researchers intended to see whether the two technologies could help address some of the myriad problems facing quantitative research, such as engagement, survey fraud and data-quality issues, according to Alexandra Kuzmina, innovation consultant at MMR Research, who led the research.

“Can technology help us prevent that data-quality disaster? We hypothesised that it can, and that it can also uncover better quality insights if people are more engaged,” Kuzmina says.

MMR Research conducted two experiments with consumers – one using an AR hologram and another using an AI avatar.

The first experiment, run through an AR app, tested three different features using the same survey, the first being an AR hologram of a researcher explaining the assignment that played before the survey, with users directed to the survey after the hologram ended. The other two elements tested a video of a human researcher introducing the survey and thanking participants, while the third was a control group using a standard survey with a written introduction.

Participants were provided with an invite to the survey and were asked to click on a link that launched the AR experience, which they could then view through their phone camera. “They were able to see the hologram appear in their physical space as if she was there in the same room as them, and creating the sense of physical presence,” adds Kuzmina.

The results showed participants in the AR group of the research provided better-quality insights, typing longer responses and including more relevant information. “We did see a significant impact on the word count for the AR cell, and a lot more meaning and a lot less junk – fewer keyboard slams and less rubbish,” Kuzmina explains.

However, there was one significant issue – the AR survey had a 93% dropout rate, with 1,478 participants entering the experiment, but only 98 completing the survey. The dropout rate for the video introduction by the human researcher was 29% and 28% for the traditional survey. “Although we did see an increase in engagement, it is not viable at the moment to burn through that much sample, at least not just yet,” Kuzmina concludes. “While video is largely accepted and adopted within our industry, AR is new to most.”

This led to the second experiment, which built on the relative success of the survey with the introductory video by repeating the experiment using an AI-generated video, with 409 participants.

The experiment again had three parts: one with an introductory video featuring an AI avatar; another with the same AI avatar video, followed by an open-ended question asked by the avatar; and, finally, a standard survey as a control.

“We needed a scalable solution, which is why we used generative AI, which is booming,” says Kuzmina. “It allowed us to generate an avatar to say pretty much anything you want thanks to real-time media synthesis technology. We hoped it would still feel human enough to increase engagement.”

The research found that an AI-written question generated 44% more words within survey responses than a standard survey question written by a researcher. There was also a 150% increase in concurrent themes, which help explain how themes that emerge in the research are connected and tell a story. “It was almost like having the AI avatar generate some sort of obligation from participants to be more helpful – not just say they like something, but actually explaining it in their own words,” says Kuzmina.

For engagement – defined as whether people wanted to complete the survey, found it enjoyable, felt their answers were important to the research and that the survey was worthy of their attention – the generative AI avatar came top, followed by the control and then the AI video asking the additional open-ended question, which came bottom of every engagement metric.

Kuzmina says that the ‘uncanny valley effect’, where robots and avatars that are too similar to humans can be more off-putting than more obviously fake avatars, could have played a role, given the increased length of time the AI avatar spoke in the second section of the experiment. “People who saw the video twice leaned more to negative impressions and responses,” she adds. “They didn’t hate her completely, but there is a tendency to lean towards a more negative domain. It is important to find the right balance. The avatar did increase engagement, and we saw deeper, more meaningful insight. But we also found too much can be weird. It is important to experiment and understand these things before jumping into something new and shiny.”

The next step of the journey will be to try new voices for the avatar and to test it in languages other than English, exploring whether there are different reactions, as well as the potential to improve research with children. But most of all, Kuzmina says, there is a need to “bridge the gap between old and new, and take participants on a journey that doesn’t alienate them”.

With generative AI growing quickly, the opportunities for further experimentation are on the horizon.

This article was first published in the July 2023 issue of Impact.