This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Find out more here

OPINION25 November 2014

The dangers of DIY research

Opinion

Companies have been putting market research studies they have commissioned from our industry into the public domain for decades. Now it’s called content marketing.

The advent of social media has done several things: it has driven up demand for such content (which is good) and it has enabled market researchers to see a wider variety of research from other companies (also good as it stretches us as an industry). However, it is also fuelling a growing trend for the creation and sharing of poor quality ‘research’, ‘studies’ and ‘reports’.

Before social media, to varying degrees, journalists were the gatekeepers, deciding which research ended up entering the public domain and which was consigned to the bin. They were mindful of quality. This was a good thing. It put a stop to the ‘around-the-office’ surveys which allegedly took place in the early 1990s. Now with the ease of self-publishing provided by blogging and social media, the flood gates seem to be wide open again. Lots of companies and their marketing agencies are shunning well-executed, insightful research conducted by market research companies in favour of cheaper alternatives. I have seen numerous examples. In fact, this type of ‘research’ is more common than good quality research from reputable sources. The six deadly sins include:

  1. DIY surveys: conducted by the client or one of their marketing agencies, with no involvement from any market research professionals
  2. Inadequate sampling: inappropriate respondents for the topic and / or not enough respondents
  3. Misinterpretation: any researcher who has helped clients prepare research data for the public domain will know that it is so very easy for non-research professionals to get this element wrong. I dread to think what errors exist within the DIY reports
  4. Poor reporting: an example recently showed a pie chart with one set of numbers, but the text described a completely different set. We all know how this comes about, but clearly the authors didn’t even notice
  5. Borrowed research: usually displayed as an infographic with a long list of different sources (some of which are questionable, like another dodgy, ‘borrowed’ infographic), which means they often compare apples and pears in terms of sample bases
  6. Lack of proper reference: some articles are based on very good independent market research, but the client / author fails to properly acknowledge the source

To me, this situation is ludicrous. There is a growing body of research indicating that B2B buyers are inclined to trust third-party content more than vendor-branded content. It seems expert opinion counts for a lot and people’s appetite for research-backed posts is high. This positions research very well to be used as content. But we must not forget that for the client, the whole point of creating content is to make their company look good, add value, educate and engage.

The approaches listed above are surely counterproductive to this aim. Even the client who has commissioned a reputable market research company but does not acknowledge this fact in their sharing of the research findings is missing the point. These approaches also completely sideline us as market research professionals and as a market researcher, I feel I cannot sit back and let this continue without passing comment.

Cherry Taylor is MD at Dynamic Markets.

8 Comments

5 years ago

This article touches on several very important points for B2B marketing. For one thing, it underscores how important it is to have independent data of high quality that can show the buyer's peer group clearly. The buyers are fairly sophisticated consumers of research and they can usually detect a sand-bagging job pretty well.

Like Report

5 years ago

I agree that many of the quoted research reports are bad but, to me, this is just a hazard of the internet. Low barriers to entry generally means lower quality and the internet is open to all. So, I think, the key question is how do you protect yourself from becoming the idiot who believes the unsubstantiated headline? Like a friendly drunk looking for a taxi home how do you find a cabby you can trust not to take advantage? RISK & TRUST The two questions you need to ask of all research is "What do I risk if this is wrong?" and "Can I trust that this is likely to be right?" If you are making decisions based on data then, how much time and money are you risking? If you are presenting to your distributors or customers or partners, then how much is your reputation worth? What will happen if this information is false? Then decide how much trust you can put in the research - and Cherry's list above is a fairly good one. Small samples, DIY, poor report detail and referencing are all good clues that the research was built from the headline down rather than up from trustworthy data. Market Research professionals don't have a monopoly on good insight but their credibility has been tested through experience and income. Like a branded cab from a well known firm they are more likely to get you safely home when your judgement may not be at its best!

Like Report

5 years ago

Thank you Hugh and Stephen for your thoughtful comments. A conversation that ensued on Twitter after this was published was focussed on what can be done. Maybe nothing, or nothing that will make a dent in the problems. I suppose all we can do is encourage clients down the right route and hope they follow.

Like Report

5 years ago

Bravo...someone needed to say this! Many of my colleagues who are in both supplier and client side positions bemoan this rush to "fast & dirty" DIY research. It may be fast, but often too dirty to be even remotely accurate. Sadly, many marketers opting for DIY don't even recognize the pitfalls. It's deemed "good enuff" because it's often quite cheap (or deemed as "free").

Like Report

5 years ago

Amen. Unfortunately, most clients don't listen to the warnings since they themselves are using DIY research and creating their own self-fulfilling prophesies. Until companies get burned publicly with this type of research, lose money, respect, and possibly lose their jobs will they heed our pleas.

Like Report

5 years ago

I think no matter what business professional are all aware of the quality of the information by analyzing the source. If its not Bloomberg or Reuter don't beleive everything you read. Despite the fact that DIY are not reliable sources, they are the most comprehensible source for the public. People use Twitter for instance to stay on top of the information, because you get the opinion of the author within 200 caracteres. I agree that most newbies in the market may be mistaken by non-professional market research reports, but tit remain a stepping stone for them to enter the market. For instance is Crude Oil going down because the middle east wants to slowdown US crude oil export or crude oil is going down as a sanction against Russia for being in Ukraine. Professional market researcher offer two different opinion despite the fact that they are professional.

Like Report

5 years ago

This is a very good article.. I completely agree with the points made. Another personal observation is clients in Asia have recruited consumer insights executives from MR firm and offered them senior level position in the organisation. These executives lack fundamental knowledge of research and do "Postman" job (Middle man between Marketing and MR Firm). I wonder how they use DIY and drive business recommendation without expertise of agency

Like Report

3 years ago

This is rubbish. If clients want to cut corners and let interns conduct poor research, that's their problem short and long term. May the best man win. In the meantime, agencies need to do more to add more value greater than the status quo. The standard survey and powerpoint deck just won't cut it anymore. Everyone has blood on their hands taking the easy route and not continuously investing and learning more about new tools, new practices, and to boot,- empowering the next gen age of researchers that have grown within and can speak on behalf of the new digitally empowered consumer.

Like Report