OPINION13 October 2011

Getting the story straight

Opinion

Esomar Congress recently heard calls for researchers to get better at telling stories and to find new ways of inspiring clients. But we need to make sure the stories we tell are grounded in reality, warns Dan Kvistbo.

Yet I worry that we are in effect discursively constructing a new research paradigm that doesn’t have substantial warrant in the real world. We’ve gone from insisting on sound methodological approaches towards simply “delivering insight” or “telling compelling stories”; from efforts to maximise the representativeness of our studies, to settling for some level of “consistency” in the sample.

It’s been some five years since Kim Dedeker – then at Procter & Gamble, now Kantar – famously said that she never thought that she “was trading data quality for cost savings” when two online access panel surveys carried out a week apart “yielded different recommendations”. A few years of industry soul-searching and research-on-research followed. The end result? Rather than meeting the inherent challenge, the market research industry has broadly responded by suggesting instead that we trade “purity” for “pragmatism” (as exemplified by our colleagues at Communispace in this article).

By and large, methodology has become a “hygiene factor” that we need not discuss as long as we excel at “inspiring” our clients – a key theme at the recent Esomar Congress in Amsterdam. Indeed at that event, the chairman and the director of the newly-renamed Dutch industry organisation MOA went so far as to define the ‘qualities’ of classic market research information providers as “distan[t], objective/representative, boring, difficult and chilly”. Compare that to the ‘new’ information providers who are perceived as “proximate, subjective/relevant, exciting, easy, fast and fun”.

While I sympathise with the view that market researchers could and should do better at communicating research results, and that there is a need to embrace new methodological approaches to complement existing methods in order to remain relevant, I’d still insist that we keep challenging the foundation of any business intelligence efforts.

We must do our best to collect and curate data that will support healthy business decisions by reflecting the opinions and attitudes of populations or target groups relevant to the respective study as accurately as possible. While this may not always be easy, fast or fun, anything less is selling our industry short.

Don’t get me wrong – I like a good story as much as the next person. But market research is still in the documentary-making business. We need to make sure the stories we tell are grounded in reality.

Dan Kvistbo is director of panels and R&D at Norstat. Find him on Twitter

@RESEARCH LIVE

9 Comments

9 years ago

I think Dan is right to point out the dilemma most market researchers have - To forgo quality for speed or to forgo speed for quality. The difference is always time and costs. On the one hand, you can have quality and tell a compelling story, but you need more time and money. You can't tell a compelling story when you have limited time and money, that is not realistic. The devil is in how you compromise quality with time and costs. But hang on! What about Technology? This should help us speed up our research processes and quality? You'll be the judge! Jasper Lim Merlien Institute http://www.merlien.org

Like Report

9 years ago

The debate isn't simply between traditional and new methods but also about how valid those traditional methods were. More traditional surveys may well have included great sampling frames but often the delivery, context and structure of the questions asked were entirely unrealistic and not representative of human behaviour. Sure the sample may have been better in the past but often the results we completely unrepresentative of how people think and behave!

Like Report

9 years ago

Thanks for the provocative posting, Dan. Not surprisingly, I agree with Steve's response regarding the limitations of traditional survey methodology. However, I don't reject the need for methodological rigor when informing certain business decisions. Companies certainly should look beyond the deep insight and great story and rely on modelling and forecasting techniques when making go/no go decisions, for example. However, the fact is that roughly 80% of new products fail, even when those traditional methods have told the brands that these products would succeed. That suggests that these methods alone are not sufficient to enable smart business decisions. Rather, to kill bad ideas early and maximize the potential relevance and success of new messages, services, products, etc., brands have to also establish a trusting, collaborative, ongoing relationship with theri customers/consumers. They have to engage in them in co-creation as well as testing, and that's where some of the tradeoffs Manila and I articulated in the Communispace paper you cited especially come into play.

Like Report

9 years ago

Great stories can only be constructed out of great data. Good, consistent data can only come out of great fieldwork. Getting fieldworkers who are at least half as educated, intelligent and experienced as the researcher is the key.

Like Report

9 years ago

I applaud Dan's perspective. For our industry to remain useful, we must base our insights and recommendations on factual data. To be relevant, we need to do a better job of interpreting data and telling a story, but if the story is based on faulty methodologies, it's nothing more than a fairy tale.

Like Report

9 years ago

Thanks for the comments, much appreciated! A few thoughts in return: Jasper; I think that there’s one option that we tend to overlook when faced with the choice of forgoing quality for speed: When the compromise on quality becomes a compromise on our professional pride and the integrity of our business, say no? Sure, we all need the revenue and we all want to help our clients the best we can - but sometimes turning down a project is doing exactly that. Technology... a double-edged sword! While it has indeed given us a range of faster, cheaper and in some aspects better methodologies, it has also provided us with false security and sometimes undermined the quality of research (e.g. ID-verification to ensure unique respondents, DIY tools in the ‘wrong’ hands, blind application of survey routers etc.). Steve; Great points, but also a little towards the “all methods have issues”-argument? One doesn’t exclude the other and as long as traditional methods account for the bulk of the global market research spend, I’d suggest that we keep adhering to best practices in all phases of a research project – including aiming for the best possible sample frames. Julie; I am very happy that you responded to my post, not least because it gives me an opportunity to emphasize that I didn’t of course intend to single out neither you and Manila, nor Communispace in any way. Merely, your post well represented how the general discourse has evolved. To your comment: Yes, traditional survey methodology has its limitations but my point is really that this is no excuse for compromising on aspects that we know affect the quality of our research. That said, I definitely agree with the rest of your points made here – or as I believe Diane has so concisely expressed it, “Never underestimate the power of n=1”. The more interesting discussion in that regard I think is where the boundaries of market research will be in the future. Branded communities are one of the “new” methodologies that are currently changing the checks and balances between the marketer, the market researcher and the consumer.

Like Report

9 years ago

Great stories start with great analysis, great analysis with great data, great data with great designs and sampling frames and so on. They are all linked to each other. Can we say any of these are more important than the others? I don’t think we can. The future of market research will be built on what we learnt from past, traditional methods. The world is changing, so is the research world. We need to adapt ourselves to this changing world by finding new ways connecting with research participants and ultimately consumers without compromising “hygiene factors”. Respondents are not waiting for us to knock their door for an interview anymore. They tell what they think about brands without being asked and words spread so quickly. That is why online communities are great tools to tap into consumers’ own world. Would it replace traditional CSS or market simulation models or brand equity studies? My answer is no. In many cases clients are not taking big decisions based on one piece of research anymore. In the new world we need to understand how all of this different information tie in together, contextualise it, see the big picture and then, tell a great story.

Like Report

9 years ago

Lots of good points but I get a bit nervous when we assert that there is a factual based 'reality' 'that can be uncovered so long as we use the right methodologies and only allow expert researchers to do the interpretation. With people being poor witnesses of their own motivations and behaviour I think this is a bold claim with a whiff craft guild protectionism about it. Things are a bit more complicated. I see storytelling as communication of findings that captures the hearts as well as minds of the intended client audience and leads to action that helps them achieve desired outcomes. Not new but something that we haven't focussed on as much as we should.

Like Report

9 years ago

Most of my points have been covered above, I guess, but I wanted to add my tuppence-worth. The difficulty that research as a profession faces (particularly when thinking about researchers in non-research organisations) is that the more we talk about rigour, statistically correct sampling and so on, the more we turn people off. Given the choice between making research interesting and engaging to clients, and making it absolutely correct in methodological and analytical terms, it's understandable that we as researchers often veer towards the more story-led approach. It's the one that works for clients, it's the one that ensures that research is used and consulted, it's the one that keeps us in work. It's not black and white, obviously; my own bias is towards understanding 'the story' whilst making the (hidden to clients) research behind it as rigorous as possible. But if it's a choice between a low-budget, simplistic, directional piece that a client will use and a larger, more correct study that will be shelved even if the budget ever materialises, I'd always take the one that's actually useful. What I'd suggest, though, is that we as researchers bring a secret sauce to the mix, namely our understanding of audiences, behaviours and dynamics. Even where we have to deviate from the theory on sampling techniques, survey design, and depth to fit within a budget or objectives, we know from years of experience which results can be relied upon, which to treat with caution, and so on. We also know how to shape a 'story' that is correct and doesn't overstep the mark of what we can prove. It's not a replacement for research rigour, but it allows us to work around such compromises and still deliver research that delivers.

Like Report