FEATURE18 July 2012
Gill Wales considers the state of report writing in the market research industry and what can be done to raise standards.
Justin Gutmann’s tone was more weary than angry. That week, he and colleagues had wasted several days trying to make sense of a succession of badly written reports from research agencies. He asked me, “Why are agencies so bad at writing reports?”
He showed me the main offenders. I could only mumble apologies on behalf of the industry. They were terrible.
Gutmann is the head of research and insight for Consumer Focus, the statutory body responsible for championing consumer rights in England, Wales and Scotland. To do this, it commissions a lot of consumer research – on the performance of energy suppliers and the postal service, public services, digital communications and financial services providers. Reports come in from a variety of agencies, including some very well known ones.
“For many clients the final report is the tangible ‘product’ they intended to possess when they signed the purchase order. If the report fails to provide a clear and reliable description of the outcome, then the client has acquired a useless product”
And while it’s not true that every report that Consumer Focus receives is poor, enough of them are to suggest there is a problem in the research industry.
Gutmann says: ‘We try and explain to agencies how critically important the report is to us as the main deliverable, as a public record of how we have spent taxpayers money” – funding comes from the Department for Business, Innovation and Skills (BIS) and from licences paid by energy suppliers and the postal industry – “and, above all, as critical input into policy and communication actions.
“But it seems to make no difference, even though our suppliers know that we publish the reports on our website – a great marketing opportunity for them I would have thought.”
To be fair to agencies, much of the work Consumer Focus commissions is complex and no one expects the reports to be perfect first time. There are always bound to be queries and requests for more detail or a rearrangement of structure. But many of the final outputs are simply not fit for purpose.
Principal policy advocate Alison Hopkins says: “We see the process as a partnership and expect to contribute to final outputs. At the same time, we rely on agencies to be professional, accurate and thorough. A poor report doubles our workload. It leads to a protracted and tedious process of double checking several iterations, and to missed deadlines that impact the rest of the organisation.”
Litany of errors
Here’s a list of some of the failings that Consumer Focus has identified. It has seen results reported in questionnaire order, rather than order of importance or interest; whole sequences of questions unreported; whole objectives ignored; incorrect interpretation of data; inconsistencies between charts and narrative; some charts with no narrative at all; contradictions in the narrative; spelling and grammatical errors; poor standards of written English; errors in product names and acronyms (including the same acronym used for different bodies – in the same sentence), and – perhaps worst of all – evidence that the report writer did not understand the subject being researched.
The problems are not unique to Consumer Focus. Nor are they new. Andrea Burton, former joint head of customer and stakeholder insight at London Underground and now an independent customer insight specialist, says: “When I first returned to clientside work 12 years ago I was horrified with the first report and tables that I received from an agency. Tables were wrongly headed, columns didn’t add up, cross tabs didn’t make sense, the standard of English in the narrative report was barely acceptable and the conclusions indecisive.”
“Clients don’t buy methods; they buy answers, solutions to problems and understanding of markets. Continued commoditisation is inevitable if agencies fail to deliver the expected value”
Some suppliers might say that this is a pointless argument to have as clients no longer want full written reports. But while a report might be unnecessary to answer a simple and immediate question, I find in my own dealings with clients that they generally do want a full record of the work: for future reference, to circulate among colleagues, to add to the body of knowledge. A set of PowerPoint slides rarely works as a standalone piece – even for people who saw the slides as visual aids in a debrief.
For many clients the final report is the tangible ‘product’ they intended to possess when they signed the purchase order. No matter how skilful the fieldwork, if the report fails to provide a clear and reliable description of the outcome, then the client has acquired a useless product. The fieldwork may as well never have been done.
Why are so many agencies delivering useless products? Consumer Focus chooses agencies carefully. Those who delivered the sub-par reports I’d seen had won the work by submitting sparkling, well-written proposals with impressive promises of quality. But there was a chasm between the promise and the final product.
Agency reaction to having this pointed out was often one of surprise. They either did not know or did not care that the ‘product’ didn’t work. One can only assume that senior research expertise had contributed to the proposals, but report writing was delegated to juniors. There’s nothing inherently wrong with that, so long as seniors continue to contribute by coaching and supervising junior staff. But it seemed as if the seniors had abandoned their juniors to struggle alone with the huge datasets these projects generated, not even bothering to read the report before it left the building. Which raises the question, how are today’s juniors going to develop their skills if their work isn’t checked and overseen by seniors?
In some cases reports were also late, implying that agencies had underestimated the scale of the task as well as the skills needed for the job. Or does the current economic situation mean they are fearful both of turning work away and hiring enough staff to deliver it on time?
Get creative
Whatever the reason, it is short-sighted. Economist Daniel Park of the MASS Consulting Group says, “In many markets, technological change enables lesser-skilled individuals to undertake tasks that could previously be entrusted only to highly-qualified, experienced, expensive individuals.
“A lot of market research can be done more quickly, cheaply and reliably than 20 years ago. The future for market research professionals is in the value they add that automated data gathering and processing cannot. Such value is found in interpretation and the conversion of analytical thinking into creative and differentiated thinking.”
But what if the analytical, creative and differentiated thinking goes only into the sales pitch and not the end product? What if research professionals consistently under-resource the work they take on?
Much has been written about the commoditisation of research and how data collection has become devalued thanks to DIY online survey tools. Some argue that the solution to this problem is to develop new, more advanced methods of research – in the hope that clients will pay more for something high-tech and shiny. But clients don’t buy methods; they buy answers, solutions to problems and understanding of markets. Continued commoditisation is inevitable if agencies fail to deliver the expected value. And ultimately, clients will look elsewhere to get what they need.
Getting it right
- Be realistic about how long it will take to write the report.
- If work is delegated, brief report writers fully; don’t expect juniors to reconstruct the nuances of the project from a couple of emails and a copy of the proposal.
- When projects are large and complex, submit a section of the report in draft for early approval. eDigitalResearch is one agency that delighted Consumer Focus by writing a single chapter in advance of the full report to check that the style and standard met the client’s needs.
- Senior executives need to check juniors’ report writing while it is in progress, actively coaching interpretation and writing skills.
- Senior executives also need to proof-read the report before it leaves the building.
- If agencies are temporarily short of report writing resources, consider drafting in a freelancer. The Independent Consultants Group, for example, has 70 freelance writers on its books, and there are plenty more out there.
Gill Wales is an independent consultant. Earlier in her career she was a market research manager for Alliance & Leicester
The world's leading job site for research and insight
Resources Group
Qualitative Research Manager– London / Hybrid working
Up to £37,000 + Benefits
Resources Group
Data Scientist (Junior) – Leading Hedge Fund
c. £60,000
Resources Group
Associate Research Director – Global Communications & Public Affairs
£50–£60,000 + 30 days annual leave + private health + other great bens!
-
Brought to you by:
©2024 The Market Research Society,
15 Northburgh Street, London EC1V 0JR
Tel: +44 (0)20 7490 4911
info@mrs.org.uk -
© Copyright 2024 Research Live
15 Comments
Ray Poynter
12 years ago
One problem is that too many clients are not prepared to pay what a good report costs. A good report is going to take several days to write. If written by a senior person, that means several thousand pounds of costs (not marked up prices, but costs). However, I agree with Gill and Justin, that a poor report is not an acceptable, cheap, alternative for a good report.
Like Reply Report
Eoghan O'Neill
12 years ago
"Here’s a list of some of the failings that Consumer Focus has identified. It has seen results reported in questionnaire order, rather than order of importance or interest; whole sequences of questions unreported; whole objectives ignored; incorrect interpretation of data; inconsistencies between charts and narrative; some charts with no narrative at all; contradictions in the narrative; spelling and grammatical errors; poor standards of written English; errors in product names and acronyms (including the same acronym used for different bodies - in the same sentence), and – perhaps worst of all – evidence that the report writer did not understand the subject being researched." Those strike me as being at very different points of the scale of importance. Spelling mistakes (including product names) and poor English (bear in mind that many international reports are written by non-native speakers) may be irritating, and indicate a lack of attention to detail, but little more. Charts with no narrative are more sinful, but once again might indicate general slovenliness rather than invalidating an entire report. On the other hand, incorrect interpretation of data is a fundamental flaw and could render an entire study useless.
Like Reply Report
Roderick White
12 years ago
This is, I'm afraid, not a new problem, though it sounds as if it has been getting worse. There has always been a big gap in quality between the best qualitative research reports and almost anything reported on in full by quant agencies. But Ray Poynter is right about costs. I suspect that competitive pressures have led quant agencies to believe that they can get away with, basically the tabulations and a few charts. It may be pretty hard to change this.
Like Reply Report
Karen Schofield
12 years ago
It's been a long while since I wrote a full report but I can't agree with Eoghan's comments that spelling mistakes and typos are trivial - they can undermine the rest of the report because if someone hasn't bothered to have their work proof-read, how do you know they have bothered exploring the data properly? Attention to detail, in analysing, interpreting and reporting on data is key, so to suggest otherwise is, I think, misguided. It's such a shame that there are agencies producing sub-standard work, basic written English should be a given for any research output! I also think Roderick's comment about the gap between qual and quant agency reports is unfair. The research type doesn't matter, it's the skill and effort of the researcher to make sense of whatever information they have, qual, quant or otherwise, and to provide their end client with something useful.
Like Reply Report
Neil Gains
12 years ago
The core of the problem is the business model of agencies, where senior time is focused on selling more research and the actual job is pushed further and further down the chain to juniors including that of report writing as the main deliverable of the work. Agencies will claim that this is the result of the push for more and more competitive costs, but the truth is that this is becoming a destructive spiral as lower costs lead to less senior time, leads to poorer deliverables, and so on. If reports are getting worse, why would any client want to pay more for rubbish? In fact, the whole business model of large agencies needs to change, and maybe changes in data collection and the rapid decline in cost per interview will force their hands. If I were a client I would be hiring good creative thinkers and planners with experience of the category, company or research approach and getting them to work with data collection specialists. The way things are going is not good for the industry, and the big agencies don’t seem to care enough to want to change. If you buy something once and it fails you can forgive, but if you keep buying and it keeps failing there comes a point when you either decide to go elsewhere or you do without it. Market research cannot continue to fail clients.
Like Reply Report
Karen Schofield
12 years ago
It's been a long while since I wrote a full report but I can't agree with Eoghan's comments that spelling mistakes and typos are trivial - they can undermine the rest of the report because if someone hasn't bothered to have their work proof-read, how do you know they have bothered exploring the data properly? Attention to detail, in analysing, interpreting and reporting on data is key, so to suggest otherwise is, I think, misguided. It's such a shame that there are agencies producing sub-standard work, basic written English should be a given for any research output! I also think Roderick's comment about the gap between qual and quant agency reports is unfair. The research type doesn't matter, it's the skill and effort of the researcher to make sense of whatever information they have, qual, quant or otherwise, and to provide their end client with something useful.
Like Reply Report
NickD
12 years ago
Yes yes yes! Or, to put it another way "no, no, no (go back and do it again)!". As you say, this isn't a new problem, and it's one that I've found across agencies and continents. If anything, the large agencies seem to have the biggest problems, and as I think has been identified, a lot of this may relate to the hierarchical nature of their structure, with senior researchers winning business and junior execs doing the legwork. But I've found it's particularly true in North America. The analysis skills and lack thereof of some senior researchers I've worked with from big agencies would embarrass a new graduate, let alone someone heading up a project. The lack of thinking about implications, priorities or what actually matters beggars belief. (I should point out as well, though, that even some excellent data-heads generally produce awful reports - it's not confined to the research-inept). It's got to the point where for every project I work on I now have to factor in several weeks of my own time to take the report, re-analyse and rewrite it - and it's a short step from there to going to a basic field and tab study. The industry is finding its margins being increasingly stifled, and this is one reason why.
Like Reply Report
NickD
12 years ago
doh, meant to add that I've been involved in some discussions about accreditation, particularly of vendors. But as I pointed out in those, an ability to analyse data and gain accreditation implies nothing about report-writing skills!
Like Reply Report
Peter Mouncey
12 years ago
Somehow I'm not surprised by Justin's findings. It is more than simply a problem within the MR sector. For 3/4 years in the middle of the last decade I ran a course aimed at junior managers in the field of marketing. I was appalled at the quality of some of the written assignments from students. They lacked any real structure, and evidence to substantiate claims; filled with detail that should be consigned to appendices; lacked a 'senior management' level summary and the use of English was lamentable (never mind the typos - using Spellcheck is not enough!); no content page etc. These were aspiring managers! When my eldest daughter was looking at universities to read English, I was struck by a poster at a Leeds open day from a leading management consultancy saying why they liked recruiting English graduates. One factor mentioned was that they could write lucidly and persuasively, with a good command of the English language. I'm also a judge for an annual competition for graduates & post graduates in marketing, and again the low standard of some entries amazes me - why do lecturers allow their students to submit poor quality entries as it reflects so badly on their institutions? Finally, it might be interesting to hear from agencies about the quality of written briefs submitted by clients............
Like Reply Report
Phil Brooks
12 years ago
Quite simply we all have to accept market research is changing, actual data collection is becoming more and more a commodity. Anybody these days can setup an online agency in their back bedroom. The key questions we (agencies) need to keep in mind are: • What differentiates us from each other? • How is our performance ultimately going to be judged? I have to agree with Karen, that a report full of typos is unacceptable. However, my take/reasoning is slightly different. If I was on the client side, I know I could look past one or two typos as the quality of the analysis is far more important alongside the recommendations that are made. That said if there were a lot of errors, it shows a lack of respect and more importantly I how could I trust the data? Having seen reports from different agencies, it is clear that for some projects the report has been part of a ‘process’ rather than the focus of the study. Here lies the challenge for agencies - how can we continuously produce innovative, creative and inspiring reports that help drive clients into action? When execs get briefed on a job, how many are trained to look for supporting data/context that is publically available on the internet? How many visit the local library or business school? How many are encouraged to challenge conventional reporting methods and PowerPoint templates? Today it’s not just important to report back what we have found, but why we have found it and do so in a way that helps drive action within a business. Pulling information from a variety of different sources is going to become increasing important for agencies in the future.
Like Reply Report