Will this do?

Justin Gutmann’s tone was more weary than angry. That week, he and colleagues had wasted several days trying to make sense of a succession of badly written reports from research agencies. He asked me, “Why are agencies so bad at writing reports?”

He showed me the main offenders. I could only mumble apologies on behalf of the industry. They were terrible.

Gutmann is the head of research and insight for Consumer Focus, the statutory body responsible for championing consumer rights in England, Wales and Scotland. To do this, it commissions a lot of consumer research – on the performance of energy suppliers and the postal service, public services, digital communications and financial services providers. Reports come in from a variety of agencies, including some very well known ones.

And while it’s not true that every report that Consumer Focus receives is poor, enough of them are to suggest there is a problem in the research industry.

Gutmann says: ‘We try and explain to agencies how critically important the report is to us as the main deliverable, as a public record of how we have spent taxpayers money” – funding comes from the Department for Business, Innovation and Skills (BIS) and from licences paid by energy suppliers and the postal industry – “and, above all, as critical input into policy and communication actions.

“But it seems to make no difference, even though our suppliers know that we publish the reports on our website – a great marketing opportunity for them I would have thought.”

To be fair to agencies, much of the work Consumer Focus commissions is complex and no one expects the reports to be perfect first time. There are always bound to be queries and requests for more detail or a rearrangement of structure. But many of the final outputs are simply not fit for purpose.

Principal policy advocate Alison Hopkins says: “We see the process as a partnership and expect to contribute to final outputs. At the same time, we rely on agencies to be professional, accurate and thorough. A poor report doubles our workload. It leads to a protracted and tedious process of double checking several iterations, and to missed deadlines that impact the rest of the organisation.”

Litany of errors

Here’s a list of some of the failings that Consumer Focus has identified. It has seen results reported in questionnaire order, rather than order of importance or interest; whole sequences of questions unreported; whole objectives ignored; incorrect interpretation of data; inconsistencies between charts and narrative; some charts with no narrative at all; contradictions in the narrative; spelling and grammatical errors; poor standards of written English; errors in product names and acronyms (including the same acronym used for different bodies – in the same sentence), and – perhaps worst of all – evidence that the report writer did not understand the subject being researched.

The problems are not unique to Consumer Focus. Nor are they new. Andrea Burton, former joint head of customer and stakeholder insight at London Underground and now an independent customer insight specialist, says: “When I first returned to clientside work 12 years ago I was horrified with the first report and tables that I received from an agency. Tables were wrongly headed, columns didn’t add up, cross tabs didn’t make sense, the standard of English in the narrative report was barely acceptable and the conclusions indecisive.”

Some suppliers might say that this is a pointless argument to have as clients no longer want full written reports. But while a report might be unnecessary to answer a simple and immediate question, I find in my own dealings with clients that they generally do want a full record of the work: for future reference, to circulate among colleagues, to add to the body of knowledge. A set of PowerPoint slides rarely works as a standalone piece – even for people who saw the slides as visual aids in a debrief.

For many clients the final report is the tangible ‘product’ they intended to possess when they signed the purchase order. No matter how skilful the fieldwork, if the report fails to provide a clear and reliable description of the outcome, then the client has acquired a useless product. The fieldwork may as well never have been done.

Why are so many agencies delivering useless products? Consumer Focus chooses agencies carefully. Those who delivered the sub-par reports I’d seen had won the work by submitting sparkling, well-written proposals with impressive promises of quality. But there was a chasm between the promise and the final product.

Agency reaction to having this pointed out was often one of surprise. They either did not know or did not care that the ‘product’ didn’t work. One can only assume that senior research expertise had contributed to the proposals, but report writing was delegated to juniors. There’s nothing inherently wrong with that, so long as seniors continue to contribute by coaching and supervising junior staff. But it seemed as if the seniors had abandoned their juniors to struggle alone with the huge datasets these projects generated, not even bothering to read the report before it left the building. Which raises the question, how are today’s juniors going to develop their skills if their work isn’t checked and overseen by seniors?

In some cases reports were also late, implying that agencies had underestimated the scale of the task as well as the skills needed for the job. Or does the current economic situation mean they are fearful both of turning work away and hiring enough staff to deliver it on time?

Get creative

Whatever the reason, it is short-sighted. Economist Daniel Park of the MASS Consulting Group says, “In many markets, technological change enables lesser-skilled individuals to undertake tasks that could previously be entrusted only to highly-qualified, experienced, expensive individuals.

“A lot of market research can be done more quickly, cheaply and reliably than 20 years ago. The future for market research professionals is in the value they add that automated data gathering and processing cannot. Such value is found in interpretation and the conversion of analytical thinking into creative and differentiated thinking.”

But what if the analytical, creative and differentiated thinking goes only into the sales pitch and not the end product? What if research professionals consistently under-resource the work they take on?

Much has been written about the commoditisation of research and how data collection has become devalued thanks to DIY online survey tools. Some argue that the solution to this problem is to develop new, more advanced methods of research – in the hope that clients will pay more for something high-tech and shiny. But clients don’t buy methods; they buy answers, solutions to problems and understanding of markets. Continued commoditisation is inevitable if agencies fail to deliver the expected value. And ultimately, clients will look elsewhere to get what they need.

Gill Wales is an independent consultant. Earlier in her career she was a market research manager for Alliance & Leicester