FEATURE1 July 2009

World of temptation

‘DIY’ online survey and analytics tools are an increasingly tempting option for conducting research for minimal cost, or no cost at all. Tim Phillips ponders the effect they could have on the industry.

?Brad Bortner, a Forrester Research analyst specialising in the market research industry, tells the story of one senior executive he interviewed recently who now describes simple customer research as “doing the monkey”. The reason: like thousands of executives globally he’s an enthusiastic convert to using DIY survey tools like SurveyMonkey wherever possible.

Bortner has been tracking the emergence of low-cost and free DIY tools like SurveyMonkey for several years, and he thinks they present a serious threat to some parts of the MR industry – unless suppliers and research departments learn to work with them. “In the last big recession,” he says, “the major research buyers fired their MR people and outsourced their work to market research firms. This time it looks like they will be trying to get their own people to do as much of their own research as possible. With DIY research there is no way to get the genie back in the bottle, so we need not to be prima donnas about this. As opposed to fighting the tool, research departments should look at how to capture it.”

This could be, as the prostitute said looking at two lovers kissing, a great profession ruined by amateurs.

The problem for the future of research as a discipline isn’t so much that SurveyMonkey has made simple research more accessible, but that the people doing the research might not be very good at it. This could be, as the prostitute said looking at two lovers kissing, a great profession ruined by amateurs.

“It’s great on one hand because it shows how much passion there is to get information about customers, but if any monkey can do a survey, perhaps any monkey will. In some cases there will be no control over design, no control over the sample,” says Bortner. At best, a bad sample might occasionally be used, or some customers might find themselves being repeatedly surveyed by different departments from the same organisation who all ‘do the monkey’ with the same address book. At worst, organisations might be flooded with misleading results and take bad decisions – and the confidence in research as a decision-making tool could ultimately collapse.

This worst case worries Sue Brooker, deputy chair of the MRS standards board. “I’m pessimistic about the future because this has the potential to derail what we do. My pessimism is that these tools are being widely used in the wrong hands. We only need a few people to get their fingers burned in a high-profile way for this to happen.
“A good questionnaire isn’t just a series of good questions. People using these tools have no idea about the tenets of good research. We know that just one word can make a difference to the result you get… Everybody thinks they can write a good questionnaire, and they can’t,” she says.

Her fears are threefold. First, that if surveys are done ad hoc by all sorts of departments across the enterprise, there is no way to track the sample, or avoid over-surveying some parts of the audience. Second, that decision-making based on this standard of research might lead the enterprise down the wrong path. Finally, the industry gets discredited by tools that were out of its control, because superficially their results look as good as well-conducted, more expensive research. “If people get too many surveys in their inbox which are badly designed, they’re going to decide that all research is rubbish,” Brooker warns.

Monkey business
It’s not just SurveyMonkey that has captured the imagination of budget-limited sales and marketing departments. In web analytics, the established businesses of Nielsen and ComScore are being targeted by Google Ad Planner, introduced in June 2008, promising the ability to “access aggregated statistics on the number of unique visitors, page views, and other data for millions of websites from over 40 countries”. Vizu offers quick and easy polling among its DIY tools, as well as its own analytics tool. Explicitly or implicitly, the message is the same: you don’t need the professionals to do this for you.

“You can’t compare a first-year sales associate with someone who has been in the business for 20 years. But will that stop people from using easy tools at the moment? No.”

Brad Bortner, Forrester Research

It’s a message that is right for tough economic times, says Bortner, in which many businesses are seeking good enough solutions at low cost. The question is, do they consider DIY research to be good enough? “You can’t compare a first-year sales associate running SurveyMonkey with someone who has been in the business for 20 years. But will that stop people from using easy tools at the moment? No,” warns Bortner. “These tools change the way that clients will think about cost. It’s similar to what online panels did to traditional research.”

That’s an evolution that Dave Goldberg, the CEO of SurveyMonkey, is quite happy to embrace. The company claims that 80% of the Fortune 500 uses its software – but he doesn’t know where or what for. Users sign up on the web – often, in Bortner’s words, “taking it and doing a run around the research department”.

He has made no attempt so far to reach out to MR professionals. “We don’t work with the purchasing department. We don’t even have a sales team. We don’t interact on a direct basis with anybody,” says Goldberg. All he sees is the employer of the user who has signed up: “For example Nielsen uses us internally. We don’t know what for, though. The reason SurveyMonkey has been successful is that it is an open platform, so it is used by professionals but also in academia, in schools, by non-profits…”

Reaching out
Eventually Goldberg would like to build bridges with MR professionals either inside or outside its clients, but you feel that he is not about to make the first move. “We want to add more of the things that professionals want, and that’s coming. But give this to an end user who is not as sophisticated as they are, and there’s no reason why you could not end up with good questions and good research too. There are many more sophisticated tools that they use, and we think that we can replace that software eventually. That means the research department can cut costs without having to cut people.”

Dan Beltramo, chief executive of Vizu – whose DIY service, Vizu Answers, helped to create the market for DIY research with customers like P&G and McKinsey – says that DIY research exists not because it is cheap but because there was a gap in the market that MR providers would not, and could not, fill. “I think the bigger issue is that there is too much bad research conducted by professionals who charge top dollar for it. One of the big advantages of DIY research is that less gets lost in translation between the client and the market research agent.

Another advantage is that because it can be so fast and affordable, it can adapt to changes in the environment or clients’ understanding much faster than more cumbersome, traditional methods,” he says. He also claims that DIY research will persist as a replacement not so much for detailed, high-value research, but for ‘gut feel’ – and as such has a positive contribution, even if the quality is variable. “

Often the cost of information provided by traditional methods exceeds the value of the information,” he adds.
However, Vizu no longer concentrates on straightforward DIY research. It offers a service which creates polls for websites, but has identified online advertising analytics through its Ad Catalyst service as offering the strongest potential growth in the short term. At the moment, this combines online tools with consultancy – but in time this too will evolve in line with the DIY model. “Our goal is to refine the technology and process so that Ad Catalyst, too, can become more truly DIY. Our growth into digital media measurement incorporates much of what we learned from our DIY service.”

Working together
So the question for the industry might not be how it holds back the tide, but how it works alongside the DIY tools that its clients are already using. SVP and managing director of ComScore Europe Mike Read recommends meeting clients to talk to them directly about DIY tools to point out where they might be giving incomplete information. He is rarely in a meeting where Google Ad Planner isn’t mentioned, he says, because clients are using it internally as a benchmark of ComScore’s service.

“Yes, the clients talk about it – one of the big discussion points when we meet them now is for them to say, ‘Your data doesn’t match our data.’ We respectfully say that, for example, the website of an English-language newspaper is read all over the world, and UK and global data are completely different, and you’re not measuring what you want to measure.”

“There is hardly a client that isn’t re-evaluating what it spends on analytics. They are all being squeezed and without doubt they are asking questions.”

Mike Read, ComScore Europe

ComScore claims that sales, so far, are hardly affected – but that doesn’t mean that clients don’t need reassurance. “Very rarely do we get a client coming to us and saying, we don’t need your tools because we get all the research we need for free. On the other hand, there is hardly a client that isn’t re-evaluating what it spends on analytics. They are all being squeezed and without doubt they are asking questions, but to date we are not seeing a move away from us – because we are happy to have a cost-justification discussion.”

Payback time
ComScore’s UK launch of Ad Metrix in May this year is part of the fightback, Read says. By demonstrating better methodology and a more useful product, paid-for research can be better value than free research. “Before, we could report on the number of pages delivered. Now we can report on the volume of advertising delivered and who it was delivered to, so in effect we are closing the circle. Free-to-use analytics give a broad indication of the usage of websites, but they’re not founded on strong research methods. They don’t provide demographic information, so it’s impossible to understand how some of their figures are arrived at.”

Outside the area of analytics, Forrester’s Bortner doubts that the cost-justification will be that easy, because the research process is being decentralised. “I have been surprised at how many large companies are now doing so much of their own work now. But I’m convinced: faster and cheaper always wins. DIY tools might not be better right now, but in time they will get better.”

Kantar’s social media knowledge leader Tom Ewing says it’s the job of the research industry to accept reality – and become an enabler, not a blocker. He sees a parallel between the uptake of DIY tools and the collaborative community-based research methods that are evolving as part of social media. Recently he floated the idea on his blog that amateurs could use an open-source process to design their surveys – posting a prototype and asking interested parties (sometimes the recipients, sometimes MR professionals) how they would improve it. The result might create better DIY surveys – and educate clients.

“The people who are in the market for DIY research usually aren’t in the market for bigger research. So surely it is up to us to educate them how to use it,” he says. “Simply they could ask if you found their questions useful, or how do you think it could improve.”

Helping clients to use DIY tools might seem like turkeys voting for Christmas, but Ewing believes it’s a grown-up response to a changing market, the sort of response made by other specialisms faced with competition from free internet tools. “In an era when knowledge is easier to come by we need to define the stuff we are happy to give away, and the stuff we charge for. We shouldn’t be acting like a secret priesthood with our heads in the sand,” he says.
Take control of how DIY tools are used, and encourage them, says Bortner, because they’ll be used anyway. “SurveyMonkey is good for what it is. You can’t do that sort of complex analytics. It’s good for a quick survey with 10 questions. Point is, a lot of surveys are like this.”



Research is just one of many industries where we can expect to see more and more products and services offered for free, according to Chris Anderson, editor of Wired and author of The Long Tail.

In his new book, Free: The Future of a Radical Price, Anderson describes how the internet has allowed costs to fall so close to zero that businesses can afford to give things away and make money elsewhere. Digital economics has turned traditional economics on his head, he argues, with the price of online bandwidth and storage forever decreasing. “It’s now clear that practically everything web technology touches starts down the path to gratis,” Anderson wrote in Wired last year.

Anderson sees a bright future for various kinds of free business models, including ad-supported content and services, free basic services supported by a minority of users who upgrade to a premium version (like Flickr), digital content that’s free or nearly free to distribute (like MP3s), and services that are free to use because the act of using them actually adds to their value (like Digg and other collaborative tools).

Free is out now, published by Hyperion in the US and Random House in the UK

UPDATE 21 July: Read Richard Thornton of Cint’s response to this article here.


15 years ago

There's no doubt that these DIY tools are here to stay, but that doesn't mean the industry should be complacent. This type of research must be treated with the highest caution - it can yield results which are at best inaccurate and at worst unethical. Market research is a professional discipline requiring expert understanding to comply with strict requirements on data protection and ethics. We as practitioners share responsibility to help clients understand the basics of good online research - which, undertaken in the right way, makes huge contributions to policy and business strategy. It can be particularly effective for engaging some hard-to-reach audiences where other tools can fail. We must not allow poor web surveys to undermine its value.

Like Report

15 years ago

I am always surprised that the typical response to the distribution of market research involves assertions of specific knowledge, practices, and statistics that are uniquely restricted to those in the market research function. Truth be told: many of the poorest quality surveys are designed by market research professionals; they are entirely too long, use respondent unfriendly question types (rankings, massive rating grids, etc.), and ask unnecessary questions. I agree the DIY tools are here to stay. But, let’s not fool ourselves into thinking that the quality of these surveys is universally lower than those created by MR professionals. As a respondent, I’ll complete a five minute survey customized from a proven MR template, but never a 30 minute survey. It seems we professional researchers forget, at times, that accuracy and quality involves not just the question types, wording, and sequencing of the questions and data analysis; it also includes completion rates which continue to plunge across the industry. DIY tools are not the enemy here; it's our historical biases toward market research. We need to innovate and change, and these tools can help.

Like Report

15 years ago

Companies are full of trained market research professionals who know how to do surveys right, yet these pros are project managers rather than mentors to the do-it-yourselfers. Instead of coordinating outside vendors, some of these pros should coordinate the inside staff who are doing research without them and help those insiders do a better job of it. Here's my call for MR departments to rethink their role. As Brad Bortner said, take control of how DIY tools are used because they’ll be used anyway.

Like Report

15 years ago

From the blogosphere: When DIY Surveys Become DYI Surveys http://tr.im/wCxT

Like Report

15 years ago

But isn't the business model of surveymonkey and the likes based on skimming the information and creating panels for onselling? Would you put your credit card information on a surveymonkey survey?

Like Report