Getting the story straight

I think Dan is right to point out the dilemma most market researchers have - To forgo quality for speed or to forgo speed for quality. The difference is always time and costs. On the one hand, you can have quality and tell a compelling story, but you need more time and money. You can't tell a compelling story when you have limited time and money, that is not realistic. The devil is in how you compromise quality with time and costs. But hang on! What about Technology? This should help us speed up our research processes and quality? You'll be the judge! Jasper Lim Merlien Institute http://www.merlien.org

Like Reply Report

The debate isn't simply between traditional and new methods but also about how valid those traditional methods were. More traditional surveys may well have included great sampling frames but often the delivery, context and structure of the questions asked were entirely unrealistic and not representative of human behaviour. Sure the sample may have been better in the past but often the results we completely unrepresentative of how people think and behave!

Like Reply Report

Thanks for the provocative posting, Dan. Not surprisingly, I agree with Steve's response regarding the limitations of traditional survey methodology. However, I don't reject the need for methodological rigor when informing certain business decisions. Companies certainly should look beyond the deep insight and great story and rely on modelling and forecasting techniques when making go/no go decisions, for example. However, the fact is that roughly 80% of new products fail, even when those traditional methods have told the brands that these products would succeed. That suggests that these methods alone are not sufficient to enable smart business decisions. Rather, to kill bad ideas early and maximize the potential relevance and success of new messages, services, products, etc., brands have to also establish a trusting, collaborative, ongoing relationship with theri customers/consumers. They have to engage in them in co-creation as well as testing, and that's where some of the tradeoffs Manila and I articulated in the Communispace paper you cited especially come into play.

Like Reply Report

Great stories can only be constructed out of great data. Good, consistent data can only come out of great fieldwork. Getting fieldworkers who are at least half as educated, intelligent and experienced as the researcher is the key.

Like Reply Report

I applaud Dan's perspective. For our industry to remain useful, we must base our insights and recommendations on factual data. To be relevant, we need to do a better job of interpreting data and telling a story, but if the story is based on faulty methodologies, it's nothing more than a fairy tale.

Like Reply Report

Thanks for the comments, much appreciated! A few thoughts in return: Jasper; I think that there’s one option that we tend to overlook when faced with the choice of forgoing quality for speed: When the compromise on quality becomes a compromise on our professional pride and the integrity of our business, say no? Sure, we all need the revenue and we all want to help our clients the best we can - but sometimes turning down a project is doing exactly that. Technology... a double-edged sword! While it has indeed given us a range of faster, cheaper and in some aspects better methodologies, it has also provided us with false security and sometimes undermined the quality of research (e.g. ID-verification to ensure unique respondents, DIY tools in the ‘wrong’ hands, blind application of survey routers etc.). Steve; Great points, but also a little towards the “all methods have issues”-argument? One doesn’t exclude the other and as long as traditional methods account for the bulk of the global market research spend, I’d suggest that we keep adhering to best practices in all phases of a research project – including aiming for the best possible sample frames. Julie; I am very happy that you responded to my post, not least because it gives me an opportunity to emphasize that I didn’t of course intend to single out neither you and Manila, nor Communispace in any way. Merely, your post well represented how the general discourse has evolved. To your comment: Yes, traditional survey methodology has its limitations but my point is really that this is no excuse for compromising on aspects that we know affect the quality of our research. That said, I definitely agree with the rest of your points made here – or as I believe Diane has so concisely expressed it, “Never underestimate the power of n=1”. The more interesting discussion in that regard I think is where the boundaries of market research will be in the future. Branded communities are one of the “new” methodologies that are currently changing the checks and balances between the marketer, the market researcher and the consumer.

Like Reply Report

Great stories start with great analysis, great analysis with great data, great data with great designs and sampling frames and so on. They are all linked to each other. Can we say any of these are more important than the others? I don’t think we can. The future of market research will be built on what we learnt from past, traditional methods. The world is changing, so is the research world. We need to adapt ourselves to this changing world by finding new ways connecting with research participants and ultimately consumers without compromising “hygiene factors”. Respondents are not waiting for us to knock their door for an interview anymore. They tell what they think about brands without being asked and words spread so quickly. That is why online communities are great tools to tap into consumers’ own world. Would it replace traditional CSS or market simulation models or brand equity studies? My answer is no. In many cases clients are not taking big decisions based on one piece of research anymore. In the new world we need to understand how all of this different information tie in together, contextualise it, see the big picture and then, tell a great story.

Like Reply Report

Lots of good points but I get a bit nervous when we assert that there is a factual based 'reality' 'that can be uncovered so long as we use the right methodologies and only allow expert researchers to do the interpretation. With people being poor witnesses of their own motivations and behaviour I think this is a bold claim with a whiff craft guild protectionism about it. Things are a bit more complicated. I see storytelling as communication of findings that captures the hearts as well as minds of the intended client audience and leads to action that helps them achieve desired outcomes. Not new but something that we haven't focussed on as much as we should.

Like Reply Report

Most of my points have been covered above, I guess, but I wanted to add my tuppence-worth. The difficulty that research as a profession faces (particularly when thinking about researchers in non-research organisations) is that the more we talk about rigour, statistically correct sampling and so on, the more we turn people off. Given the choice between making research interesting and engaging to clients, and making it absolutely correct in methodological and analytical terms, it's understandable that we as researchers often veer towards the more story-led approach. It's the one that works for clients, it's the one that ensures that research is used and consulted, it's the one that keeps us in work. It's not black and white, obviously; my own bias is towards understanding 'the story' whilst making the (hidden to clients) research behind it as rigorous as possible. But if it's a choice between a low-budget, simplistic, directional piece that a client will use and a larger, more correct study that will be shelved even if the budget ever materialises, I'd always take the one that's actually useful. What I'd suggest, though, is that we as researchers bring a secret sauce to the mix, namely our understanding of audiences, behaviours and dynamics. Even where we have to deviate from the theory on sampling techniques, survey design, and depth to fit within a budget or objectives, we know from years of experience which results can be relied upon, which to treat with caution, and so on. We also know how to shape a 'story' that is correct and doesn't overstep the mark of what we can prove. It's not a replacement for research rigour, but it allows us to work around such compromises and still deliver research that delivers.

Like Reply Report