OPINION25 May 2012

Big data? Big problem

Most research companies are barely scratching the surface of the big data challenge, says Tim Macer.

The Economist recently ran an article on “Big Data” in its special report on International Banking. Its assessment of banking is of an industry surprisingly resistant to embracing the internet as an agent of change in banking practice. It reveals, counter-intuitively, that the number of bank branches has actually risen by 10-20% in most developed economies – yet most customers pass through their doors once a year rather than once a week, as used to be the case.

The newspaper explains this paradox thus: banks with a denser branch network tend to do better, so adding more branches is rewarded by more business. But it’s business on the bank’s terms, not necessarily the customer’s. It does not increase efficiency – it increases cost. And, as The Economist points out, banks’ response in general to customers using mobile phones for banking has been lacklustre, even though customers love it and tend to use it to keep in daily contact with their accounts. It’s a level of engagement that most panel providers would envy.

All of which is to say that there are parallels here with our own industry. Here at Meaning, we’re just about to release the findings of the latest annual MR Software Survey, sponsored by Confirmit. In a sneak peek, Confirmit blogger Ole Andresen focuses on an alarming finding about the lack of smartphone preparedness among most research companies.

But what interests me is the Big Data – both in The Economist’s report and our own. The former offered a fascinating glimpse into the way banks were using technology to read unstructured text and extract meaning, profiling some of the players involved and the relative strengths of different methods. This is technology which is improving rapidly and can already do a better job than humans.

In our survey we asked a series of questions on unstructured text. Research companies – in embracing social media, “socialising” their online panels and designing online surveys with more open, exploratory question – are opening the floodgates to a deluge of words that need analysing: at least, that was what we suspected.

It turns out that half of the 230 companies surveyed see an increase in the amount of unstructured text they handle from online quant surveys, and slightly more ( 55%) from online qual and social media work. Yet the kinds of text analytic technologies that banks and other industry sectors now rely on are barely making an impact in MR.

Most research companies are barely scratching the surface of this problem. It’s not the only area where market research looks as if technology has moved on, and opened a gap between what is possible and what is practised. There’s much more on this in our report, which will be publishing in full on 30 May. Highlights will also be appearing in the June issue of Research magazine.


1 Comment

11 years ago

Nice article. It seems after this year's AAPOR conference there is a huge amount of buzz about mobile surveys and data collection. I'm doing my best to cover this on my blog: www.MarketResearchTechnology.com and connect the dots. Mark

Like Report