As the LLM rubber hit the road, ‘game-changing’ started to look different. Pitch decks collapsed meekly under client-side follow-up questioning.

Our attentions now turn to more serious forms of AI, those more aligned with the realities of insight work. Here, symbolic and neuro-symbolic approaches combine formal reasoning with machine learning. As such, we expect this next generation to find a warmer reception than did a translation engine pretending to think. In fact, we can’t wait.

We’ve also a sober idea of what to look for. The failure of AI’s three headline promises was instructive in defining what this next generation will need to do differently, to recapture both imagination and budget.

Consistency

Some claim that synthetic data is up to 80% accurate. ‘Reason to believe’ or an embarrassing admission? You decide.

Being 80% accurate on each survey question is not the same as being 80% accurate on the survey findings overall. Error compounds. Three measures that are each 80% accurate combine to a coin flip. A series of independent point estimates is not insight. Even the most basic debrief draws on well over a hundred coherent measures.

Proponents quickly pivoted to AI’s excellence at anticipating the broad skew of real data, something researchers never needed help with. We already have a strong sense of skews before fieldwork begins, and once we aggregate, say, the top two boxes, back-of-a-fag-packet guessing takes us comfortably beyond 80%. That is, if guessing aggregate survey answers were a pub quiz, we would win every round. This has never been the hard ask.

Representation

Synthetic data performs best where data is plentiful and homogeneous, and worst where it is sparse, heterogeneous, and context-dependent. Yet it is precisely within harder-to-reach minority groups that it has been most enthusiastically positioned as a substitute, exactly where wide data collection matters most.

For example, a West London minority community is not interchangeable with communities elsewhere in the country, nor with more dispersed populations. Treating minorities as a single box to tick, whether by perturbing, imputing, or upweighting existing data, adds no new information and quickly returns us to coin flip territory. Such is the nature of diversity and of maths. The instruction that typically accompanies synthetic boosts tells us everything we need to know about how defensible the data are: “Please use sparingly, and only if you must.”

Prospection

Our hope is that AI ceases to be retrospective. To be useful, AI must understand and envisage changing contexts. Here we were assured that “things will only keep getting better”. In practice, the opposite happened.

Faced with the haemorrhaging costs of inference, AI providers relied on optimisation strategies such as compression, attention thinning, and lower-cost inference paths. Rather than formally reducing context windows, tokens are still accepted but not processed with the same fidelity. The result is that the power which drew a standing ovation two years ago is no longer available.

You need not take my word for this. Test by feeding your favourite model progressively larger volumes. In my own experiments, quality begins to degrade somewhere between 800 and 1,000 words. Add more, and omissions, distortions, and confident misreadings are the norm. And sometimes it works. Sometimes it doesn’t; as a feature, not a bug.

What generative AI has done, unintentionally, is remind us from where insight springs; our value is not what AI developers presumed. We aren’t here to produce spot estimates or to condense what is already known. Strip away the consultancy gloss and, whether qualitative or quantitative, the core of market research is understanding variance among real customers. Sometimes that variance is conceptual, surfaced through language, metaphor, and meaning. Sometimes it is numerical, expressed in distributions and intervals. Neither is amenable to guesswork.

The mistake was assuming that ‘good enough’ is a low bar. That client-side apathy to the detail ran thick enough. Until the next generation of AI appreciates how hard we work the data we do use, it will remain impressive and fast, easy to cheer on, but fundamentally beside the point.

Ryan Howard is head of analytics at Rigour Research.