FEATURE5 February 2014

‘Big bang’ disruption – and five keys to survival

Like it or not, market research is being disrupted. To survive, research companies need to stop talking about value and start delivering it, argues Ipsos’s JD Deitch.

Res_4011192_big bang

The practice of market research has surely weathered its share of changes over the past half century. From mail panels to online communities, warehouse checks to barcodes, surveys conducted via paper, telephone, the web and mobile devices, to new neuroscientific approaches, practices have evolved in ways few could have ever imagined. Advances in technology, our inundation in data, and – perhaps most importantly – the business world’s increasing reliance on insights over intuition have significantly changed the industry.

These changes have dramatically impacted research firms. Companies are scurrying to develop new products around mobile, big data, and social media without cannibalising their legacy business. At the same time they are grappling with technological and operational innovations that challenge the foundations of their business and the capacity of their organizations to execute.

The business literature would suggest that the research industry, like so many others in the digital age, is being disrupted. In their 1995 Harvard Business Review article, Joseph Bower and Clayton Christensen described this as the arrival of new entrants to a market who, with cheap or inferior substitute goods, would start picking off incumbent firms at the low end of the market. These substitutes would gain enough traction to turn the eye of core or more profitable customers. Incumbent firms would be forced to follow or lose share. One need look no further for relevant examples than the way the research industry moved online at the turn of the millennium. Recall how many looked down their noses at the “inferior” methodology that sacrificed quality for speed and price. Today’s DIY research and sample selection tools are likely yesterday’s online research.

Expanding on this theory, Larry Downes and Paul Nunes wrote of more seismic shifts (May 2013, also in the HBR) that they termed “Big Bang” disruptions. If “normal” disruption threatens a product, a Big Bang can bring down whole product lines and even entire markets. They cite the market for GPS devices, which, two years after Google released its free mapping software for smartphones, is on its knees.

I believe the research and insights industry is in the midst of a Big Bang disruption.  The rest of this article presents causes and attending symptoms behind this belief and concludes with five themes traditional providers will need to adopt to survive.

Clientside drivers

Ironically, one of the more challenging developments for the industry has been its clients’ increasing reliance on insights, to drive strategy and measure performance. This, coupled with the buzz around analytics and new techniques like SoLoMo, has legitimately piqued curiosity across the organisational chart, creating new consumers of data from the C-suite down to line management. This should be good news for the industry, yet there are dark clouds on the horizon.

Esomar recently released estimates of spending on insights from 2010-2012 (see below). Most interesting are the different trajectories between spending on “traditional market research (MR)” versus new techniques. Traditional research has grown by 2% cumulatively over the three year period. Given the soft business climate, the numbers aren’t bad. The market for new approaches, however, in which Esomar includes online communities, web traffic measurement, and research reports, has a compound annual growth rate of 23%. Notably, the new Esomar estimate includes neither online analytics nor social media research nor management consultancies. Of these, one would suspect at least social media work to be sharply on the rise. 


In 2010, for every dollar spent on traditional MR, 44 cents was spent on new approaches. In 2012 it was 65 cents. This steep increase cannot last. Clients know that they are spending more money on insights. The obvious question now is, “What they are going to do about it?”. The less obvious question is, “How will this impact traditional MR companies?”.

Seeking value
Legacy firms should be concerned by the Esomar figures for two reasons. First, they suggest there is little opportunity for revenue growth in traditional products. This isn’t intrinsically surprising given the maturity of this part of the market and the known value of its techniques. While traditional researchers can bemoan the fact that new products don’t face the same reckoning around ROI and methodology that older products do, it seems safe to conclude that this shift in spending is real.

“The insights industry may not be in a world today where consumer-like customisation is possible, but this day will surely come”

Jonathan Deitch

Jonathan Deitch

Second, the figures suggest there are new dollars on the table. The Esomar numbers specifically challenge the old industry saw that clients’ search for value is just a thinly-veiled ploy to negotiate price. In the days where research was a line-item expense housed in a single department, this was unquestionably true. With data permeating client organisations and money being spent on insights embedded elsewhere in the P&L, this excuse grows more tenuous every day. 

Firms should expect that clients will recognise this and seek to rationalise expenses on insights. Ordinarily this would require clients to make meaningful tradeoffs and accept limitations. But consider their daily experience as consumers where they are allowed – even encouraged – to customise the products and services they buy. If they want Feature A and not B, then not only do they get A and not B, they only pay for A. For the most part, insights firms don’t work this way. The industry’s practice of “bundling” is often little more than a juxtaposition of rigid products that are very complex and priced to support a significant operational cost base. Knowing this, clients (or their procurement departments) put pressure on firms to show line-item costs and to deliver only what they find useful. The insights industry may not be in a world today where consumer-like customisation is possible, but this day will surely come.

In the final analysis, the above factors point to increasing headwinds for traditional firms. At best, they face bleak growth prospects. At worst, they will come under significant pressure as spending shifts.

Research is a speed bump
The other broad theme, the one that heralds the Big Bang, has to do with the way the industry approaches engagements. Despite efforts to deliver relevant insights that drive action, the delivery of those insights is an end for the firm. For the client it is only a means, a speed bump on the road to execution. The speed bump metaphor is indeed an apt one. Traditional research is often difficult to use. It suffers from unquantifiable bias and annoying imprecision and thus requires initiates to interpret results. One could joke (grimly) that the fate of traditional research companies and methodologies is directly tied to that of aging corporate researchers.

“Customer data has taken hold outside its traditional home in the marketing department. What this means for traditional MR companies is that the audiences who supported them are losing – or have already lost – their monopoly on budget and methodology”

The fact that customer data has taken hold outside its traditional home in the marketing department has accelerated this trend. This started with customer centricity models, where sales and financial analysts began to look at market segments to quantify their profitability. It spread naturally to product development. Now it is essential to distribution across both legacy and connected channels. More people are relying on data than ever before.

What this means for traditional MR companies is that the audiences who supported them are losing – or have already lost – their monopoly on budget and methodology. Their newer interlocutors do not want to be lectured on sampling or questionnaire design. They are at the sharp end of the spear. They want information that helps them execute crisply. Their needs strike at the heart of legacy MR business models. Firms can lament the situation, but at some point – perhaps as soon as tomorrow – someone will find a way to meet their clients’ needs with reliable data that cuts them out.

One can properly view the threat of Google’s consumer surveys and dashboard in this context. Google is not trying to out-research the industry, nor do they appear to have specific designs on the market. Their original single question survey (now ten questions) isn’t a cheap substitute and their inferred demographics will end up being good enough. Google is doing this because monetising content is its business model. At the same time they are learning more about people, which contributes to their success. They will find new and clever ways to deploy these tools that enable their clients to more effectively execute their targeted customer acquisition strategies. They’re not going to be asking research firms for help.

Though the desire for insights to drive strategy and tactics has never been greater, in the rooms where final budget decisions are made, research is – still – too often regarded as a necessary evil. Businesses want profitable outcomes as quickly as possible. Do they need to operate at light speed? Very few do. But when they get an idea, they want to go and execute without delay. The insight is a means to this end, one that is desirable but ultimately one that is negotiable, substitutable, and even avoidable.

New skillsets, new organisational designs
Already buffeted by uncertainty in their own markets, the industry’s savvier clients are starting to retool their organisations to chart a new course.

One way they are doing this is by bringing on new specialists. New techniques, especially those that involve “big data”, require new and different skills. The “researcher” of tomorrow will need to possess both legacy skills (like traditional statistics, survey design and sampling, and econometric modeling) as well as new skills (Bayesian models, data management, and analysis at a massive scale). She or he will need to understand their strengths and weaknesses and know when each should be used.

“Clients are also experimenting with new organisational designs where they invest a C-level employee with “data czar” or insights responsibilities. The jury is still out on how effective centralisation will be”

Clients are also experimenting with new organisational designs where they invest a C-level employee with “data czar” or insights responsibilities. Whether this resides with the CIO or CMO or the new CDO (chief data officer), the goal of these designs is to expose and rationalise the organisation’s data needs. The first two tasks on this person’s list are typically to oversee the increased spending that has arisen from new insight consumers and then assess the IT implications associated with storing and disseminating the data.

The jury is still out on how effective centralisation will be. Many organisations have information power centers that may resist this change. Centralisation has its own downsides which must be weighed against the benefits of coordination. Nevertheless, the implications are profound for suppliers: exposure and rationalisation of expenses means a greater likelihood of zero-sum decisions that limit revenue growth.

There are every-day realities which suggest the industry has time to react. Their clients each have a limited willingness and capacity to handle change, and some execute better than others. But make no mistake: all of them – and all the companies seeking to sell them insights – are trying to crack this code.

Environmental factors

In addition to clientside drivers, there are a number of environmental factors that make the search for insights – survey-based especially – increasingly challenging.

Foremost is the experience that real people, the participants whose behaviors and beliefs generate the insights, must endure. Innumerable inches have been written on the merits of genuine engagement and improving the participant’s journey, yet the experience still largely stinks. Technology is only making this worse. Firms automatically broker people in real time: a participant from provider A can end up in a router (an automated respondent allocation system) for company B taking a survey for company C. In this world of joint consumption, the outcome is a “tragedy of the commons” where each company’s individual pursuits spur a race to the bottom and nobody looks after the interests of the people whose responses drive major business decisions.

Once, the industry’s core value proposition was that it gave ordinary people a voice, a direct line to their favorite brands and retailers. The industry had a virtual monopoly on this, one which social media has completely destroyed. This, combined with the shoddy experience, explains why responses rates for online panels have declined. The transactional relationship between firms and panelists (‘give us your data, we’ll give you a reward’) creates no incentive for loyalty. Indeed in today’s connected world the experience is banal next to more genuine alternatives. The rise of proprietary “insight communities” bears this out. The companies who furnish these communities gain leverage precisely because of the member’s affinity for the brand or retailer sponsoring the community. Whether this trend will continue is uncertain. What’s sure is that respondents make instant choices about whether they will participate, and only the companies offering real value will earn their time.

Five keys to the future of insights

The industry’s reaction to the changes steaming through it has been to treat them as if they were the “normal” kind of disruption. To wit, firms have been chasing insurgents to launch new products alongside the old. This sort of marginal thinking will not suffice, especially when new entrants don’t have sunk costs that limit their actions. To become leaders in the new world of insights, firms will need to do five big things.

1. Make money for their clients
The industry talks a lot about delivering value, yet continues to pull up short of the finish line, as if the delivery of the presentation was the end of the road. Tomorrow’s winners will actively participate in their clients’ execution. Firms with industry specialisations will find this easier to do as their people often have authoritative operational experience. Insights producers that are corporately connected to teams that execute (via partnerships or as subsidiaries of a vertically-integrated conglomerate) will need to capitalise on these relationships. Despite the recent wave of consolidation, the promise of this sort of seamless integration appears to be still one of prospect.

2. Build their own assets
In an era where investment tends to be mainly in technology, companies that differentiate themselves with a broad portfolio of real assets will create competitive advantage.

This starts with human assets. For the industry to truly make money for its clients, the people who build the products, process the data, craft the deliverables, and sell the services will need to have walked the walk. They must understand the trends and have had compelling strategic and operational experience.

Firms will also need real data assets. Whether online or offline, transactional, behavioural or attitudinal, proprietary data sources are a point of significant differentiation. Companies that own and can exploit data to become their clients’ currency build formidable barriers to entry.

3. Breed natural quality by acting considerately and with integrity
It is impossible to understate the danger the industry faces from the miserable experience and cavalier approach to privacy to which research participants are subjected. The wounds are self-inflicted, the costs are real, and the blame is easily spread across new and old techniques. Perhaps for no other reason than because they must, firms will develop processes and practices that are inherently forthright and respectful across a range of issues.

“It is impossible to understate the danger the industry faces from the miserable experience and cavalier approach to privacy to which research participants are subjected”

This starts with clear and explicit policies around privacy and data use. Security and privacy will, for the foreseeable future, remain top-of-mind for both companies and research participants given the mystery that shrouds electronic data collection and the suspicion this engenders among real people. Indeed the industry would be wise to proactively get its house in order lest it lose any ability to influence the debate to regulators and consumer advocates.

Engagement fundamentally becomes easier when participants understand how their data will be used, but that’s only part of it. Again because there is little choice, firms will genuinely engage with participants because, like brands, their success is predicated on authenticity. At a bare minimum, actively-generated insight “transactions”, (i.e. those that involve questionnaires and interviews) will be well-thought out and considerate of the participants’ time. Companies will deliver tangible and intangible value for participation, in their own currency or in others’ and in doing so reinvent the idea of panels. The data clearly show that building participant loyalty is still good business for research suppliers.

Finally, to paraphrase Shakespeare, quality will out. While concerns about methodology often fall on deaf ears, there is growing evidence of the care needed in drawing conclusions in today’s fast-moving environment. There is excellent work being done on techniques and methodologies that improve accuracy and reliability so businesses can feel confident making decisions. Wasteful investments will drive their adoption and reward their exponents.

4. Ooze operational efficiency
This change is already underway. The survey research business is already horizontally integrated. It is possible to program questionnaires, connect sample, and deliver data in a single system. The usability and flexibility of these systems is improving rapidly. Indeed, not having an integrated platform will soon be the exception, and rightly so. While speed isn’t always essential, companies that are clunky in their execution will be at a competitive disadvantage on speed and cost since efficiency will be attainable without compromise. We are already seeing the next, more ambitious step in this transformation. These are “insight platforms” that enable the combination of (or at least the juxtaposition of) information from multiple techniques or that otherwise connect the insight to the execution while it’s in flight.

The potential benefits from a platform strategy are huge. Platforms create significant operational efficiencies through common tools, processes, and protocols. Properly designed, a platform renders one virtually agnostic about the type of approach used to gain the insight. If it’s got data, it goes into the system and comes out the other side in one holistic and aesthetically-pleasing view to the client. The most effective platforms will create revenue streams for their owners through licensing arrangements to companies who need the tools but lack the resources to build their own.

5. Find new models to achieve profitability
The most important thing that will distinguish tomorrow’s leaders is their approach to managing the P&L. Because they’ll have made investments in people and platforms, they will be able to invent new ways of delivering cohesively on their clients’ total needs. Tomorrow’s product managers will work hand-in-glove with operations and IT to build solutions that feed their clients’ execution. They will acquire the right amount of data and not over-collect simply because digital storage is cheap or because they (or the client) decide to “just throw those questions into the survey to see what we get”.

Winners will become methodology agnostic, emboldened by the teams and the tools that enable them to think broadly and act efficiently. Those who attempt to specialise may be able to carve out positions of strength in the market, but they will certainly not be exempt from the need to have top-notch solutions that enable effortless delivery.

This is not a call for the industry to adopt business models that rely on entirely unique solutions per engagement. It is rather a belief, driven by daily observation, that the days of the industry’s current models are limited. Flexibility will come when firms think “outside in”. Client-first thinking will create virtuous circles ensuring that firms innovate without overengineering solutions or creating needless cost.


The business world is at the precipice of an uncertain future where the consumer’s connectedness is turning traditional strategy and execution on its head. This is creating existential problems for brands and retailers, leaving industries that serve them caught in the wake. As the insights industry tries to cope with these changes, the biggest risk it faces, as Christenson reminds us, is assuming the future looks like the past. To decide on appropriate action, firms must avoid thinking marginally when far more profound actions are necessary. To succeed, they will need to envision the business they would build if they were starting from scratch.

Jonathan Deitch (@JDDeitch) is senior vice-president for respondent access and engagement for Ipsos


11 years ago

In their book, Downes and Nunes compare the 4-stage evolution of big bang disruption to the shape of a shark-fin starting with a singularity. Given Google Consumer Surveys launched 2 years ago, what phase of this model does the author believe we are in now?

Like Report

11 years ago

Ben, I've only just picked up a copy of the book. It's in my reading queue and I've got 2 books in front of it. I'll have a look and respond when I get there. - JD

Like Report

10 years ago

You certainly drive home some very good points about the challenges the marketing research industry is changing. I'd just add that some of the challenges the industry faces originate in false claims about what is new and different. I've included a link to a blog I wrote a few months ago that addresses this issue. I welcome your reactions, and those of others who read this. http://www.maritzresearch.com/soundcheckblog/industryresearch/emperor-wearing-all/

Like Report

10 years ago

Great article! In some ways the current state seems more like the 1950s where we knew little about consumers and how they spend the dollars, albeit a different reason this time with too much data as the culprit. We don't live in an ordered world where our desks are in rows in the same room; we live in a world where our desks aren't even in the same country and nobody stays still for very long. When you practice science, especially chemistry and physics you learn quickly about randomization and kinetic energy. The interaction of atoms becomes the interaction of molecules and those interactions continue until exhaustion occurs. The ability to intersect is driven by the size and shape of the container and the method of introduction. Thus, even in the same container, their is some differentiation in each experiment and an exponential number of interactions. Now imagine the world as the jar and communication as the catalyst. The communication variable is infinite, not finite and continuous. That's why dashboards have become a critical part of the landscape and why real time analytics are critical. Traditional MR is only effective where the universe of response is known. Where you have control, i.e. product testing, advertising and such MR is still a great tool, but it's not a great predictor of consumer behavior and to be honest it never was - it just didn't matter so much before. The smart move for MR is to stick with what you can control as a finite set and use BI, Big Data and such as a corollary universe measurement of the impact of your activity and its impact on target segment behavior. Quit trying to boil the ocean!

Like Report