The research surveyed a total of 7,755 participants across seven different sampling services to assess the platforms’ effectiveness, and compare the results with those generated by an AI platform using synthetic respondents to simulate survey responses.

The sampling services included two opt-in panels, microtask platform Prolific, two multi-source aggregators, and two river samples recruited from Facebook and Instagram, while the synthetic audience benchmark was provided by AI platform Electric Twin, which also provided funding for the study.

The research team, led by Michael Muthukrishna at LSE and Ben Warner of LSE and Electric Twin, found that there were substantial variation in data quality, demographics, and reported attitudes and behaviour, despite identical quotas and survey instruments.

However, the study said that beyond attention and data quality, “no single platform emerged as superior on all fronts”.

The study suggested that differences between platforms largely arose from sample composition rather than response behaviour, with platform choice having implications for replication, generalisability and the validity of marketing insights.

For example, there was a 10 to 20-point percentage swing in political voting estimates between platforms, which would be enough to influence different decisions depending on the choice of platform.

Synthetic data could capture “central tendencies similar to real respondents”, the research paper said, which it suggested means large language model-based research could be a useful way to carry out research prior to research with humans.

Respondents were asked about topics such as financial outlook, shopping habits, social media use and the UK’s support of Ukraine, with data from each platform was collected and weighted with the same methodology and recruitment targets, using age, gender, education and voting history for comparability.

Low-quality responses from bots, duplicate accounts, or respondents who failed attention checks were removed from the sample. 

Dr Michael Muthukrishna, professor of economic psychology at LSE, said: “This study offers some of the first solid, large-scale evidence that a synthetic audience can stand in for a human panel and get you to the same place. Across most of the questions we tested, Electric Twin’s simulated respondents landed within the same range as the responses of real people.   

“That’s a significant finding for anyone doing research under time, budget or practical pressure. Synthetic respondents don’t fatigue, don’t forget and don’t give in to social bias.”  

Dr Ben Warner, visiting senior fellow at LSE and co-founder at Electric Twin, added: ”This study shows that at real scale, with independent oversight, a synthetic audience produced results that were consistent with human panels across most of what we tested. That’s a strong indicator that this approach can be trusted for exploratory and iterative audience research. 

“It also puts synthetic audiences in useful company. This study did not set out to proclaim one research method as the best – it set out to test methods properly, side by side, at a scale the industry rarely gets to see. Synthetic audiences earn their place in that conversation by being testable, repeatable, and inspectable in ways traditional sampling cannot match.”