Panning for gold in big data

In fact, unstructured data is reported to represent around 80% of all organisational data, and is often where real gems of knowledge remain dormant.

Sections of market research consistently maintain that humans evaluating verbatim data will always produce more accurate results than automated analysis. Whether or not this is true, the sheer volume and complexity of data now being captured is prohibitive to using manual coding alone. The reality is that research data now presents itself in ever-increasing volumes of free-form content – not only from survey responses but also from sources such as call centre records, social media comments and customer service feedback.

Extracting key insights from ‘big data’ is akin to panning for gold, with small nuggets hidden among a vast amount of less valuable content. Sifting through all this content is time consuming but critical to get an accurate view of respondent feedback.

Quick response

A brand’s ability and speed of reaction to its customer base is increasing in importance, as brands look to promote and maintain a positive image in the market.  Forward-looking MR companies therefore need to respond to growing calls from clients for improved social and text analytics capabilities, and to understand the real business value of integrating social insight and analysis of unstructured text with MR and VoC programmes.

There are many reasons why most market researchers have, so far, continued their focus on structured data: it is core to existing research practise, it is far easier to collect, measure and analyse. More importantly, capturing and analysing unstructured feedback from certain sources, and particularly social media, can often present a challenge to the codes of conduct to which MR organisations must adhere.

We are already seeing a small number of MR organisations taking on the challenge of text analytics, and some who even consider text analytics to be the natural successor to traditional coding.  These organisations are moving beyond the realms of traditional research and using specialist technology tools that allow them to define the categories of content relevant to their clients, automatically determine sentiment within each piece of content, and then correlate insight using the same categorisation and sentiment model across all research channels.

Meaningful methodology

Of course, technology is only one part of the solution. While software can solve the challenges behind collecting, identifying, coding and analysing full text content from a variety of sources, doing something with the resulting data is where MR agencies can really gain competitive advantage. This is where methodology comes into play: by partnering with the right provider, market researchers are perfectly placed to help clients to identify relevant data sources, determine scales of measurement and present data in meaningful formats.

By integrating tailored software tools and adapting their proven research methodologies to suit the complex requirements of big data, some MR organisations have already demonstrated that analysing and interpreting unstructured data can add commercial value not only to their clients’ programmes, but to their own bottom line.

This is because the nature of social media content, in particular, and the level of insight it can present, adds richness and context to existing research data that simply cannot be obtained through traditional feedback channels.

The example of the few should be a message to the many: MR companies are ideally positioned to help businesses understand, evaluate and act upon text-based feedback from almost any data source. Including analysis of social media and other unstructured data within their research capabilities will enable them to keep pace with the rapid growth of big data, and place them one step ahead by extracting meaningful insights from it.

The MR agencies who are able to maintain and strengthen their position as providers of measurable, actionable insight will be those who can take the role of market research to the next level. 

Wale Omiyale is senior vice-president market research, Confirmit.