FEATURE1 December 2009

Promises and pitfalls of social media

When Harris Interactive began developing a solution for harvesting and integrating the growing amount of unstructured data available to researchers, we looked at traditional sources such as open-ended survey questions and interview transcripts, but we were especially intrigued by the promise of a new source of market insights: mining the abundance of unsolicited user-generated content on the web.

All of us who use the internet are already consuming the fruits of social media. Rejecting Facebook or refusing to join LinkedIn does not inoculate you from the social media experience. A Google search that calls up links to Wikipedia or blogs shows that you are enmeshed in the wonderful world of user-generated content. Watch a video on YouTube or browse family photos on Flickr and you are being touched by the pervasiveness of a type of media that is socially constructed and highly networked.

This ability to create and share is part of an information revolution. The expanse of available content for researchers ranges from the more archaic list servers and bulletin boards of yesteryear to the ephemeral staccato of Twitter. Anywhere a public comment can be posted is a source for the researcher sweeping the web.

For most practitioners, the initial promise of this expansive monitoring capability is tremendously appealing – an appeal that is often based on the following beliefs:

  1. Cost: Content-rich social media is free, and can be accessed in an automated and repeatable way. This means that social media is less expensive than traditional research.
  2. Accuracy: Given the advances in text analytics and machine learning we should be able to accurately identify what people are discussing and the emotional disposition of those conversations.
  3. Scope: There is so much content on the web that surely one can find narratives on almost any topic or perspective.
  4. Representativeness: Given the census-like approach to collecting content from the web, any analysis should be a more accurate representation of market experiences than other methods – so sampling or weighting will no longer be necessary.
    In reality it’s rather more complicated than that. Let’s look at these four beliefs in turn.

Cost ā€“ a free lunch?
Generating defensible analysis from social media content has an important upfront cost. While many of the services and tools offering complete solutions appear inexpensive, the small price tag is often restricted to online self-help kiosks that are only one step removed from Google Trends. Meeting the basic industry standards for market research means absorbing the significant costs of all the steps needed to ensure valuable and reliable insights. This means investing in the following processes:

  • thoroughly testing search and source parameters
  • building categories that map all the content
  • validating and tuning the volume and sentiment measures used to identify trends
  • converting this wealth of detail into something that is meaningful for decision-makers.

Beware if your platform or service does not give you the ability to intervene at each of these steps. The exponential nature of social media data collection has serious implications for error propagation. In other words, mis-specified search parameters or ill-conceived category and sentiment definitions can have a disproportionate impact on the interpretation of results. But with the successful achievement of each step comes the promise of automation and re-purposing designs, and this does translate into considerable cost savings for ongoing projects.

Accuracy ā€“ are you sure you understand?
The success of any social media initiative depends on the ability to accurately summarise what people are saying and determine the extent to which these conversations are negative or positive. Today there is a considerable divide between what many social media monitoring services pass off as analytical capabilities, and what really constitutes state-of-the-art text analytics.

Mapping the thematic structure of social media is a messy business that presents real problems. Platforms that do not offer a natural language processing capability should be avoided because counting words that are not linguistically connected does not accurately reflect meaning. Similarly, platforms that only provide document-level summaries often blur the rich interplay of themes and ideas articulated throughout conversations. Understanding linguistic connections between words at the sentence level is essential for untangling the complex interweaving of thoughts that give us the clues about what people think and feel. These are the essential building blocks of a defensible categorisation. The ability to tune sentiment is also essential for achieving anything near the 80% accuracy threshold found in marketing pitches.

One of the exciting promises of social media monitoring is the timely way in which unknown or newly emerging patterns can be identified. At the very least, platforms with simple clustering algorithms that can handle binary linguistic relationships offer a simple way to identify emerging or unknown categories – critical for troubleshooting in brand and reputation work.

Scope ā€“ have you covered all the gaps?
The fact that we are still far from the ideal of the semantic web means that search tools are not based on natural language. The implication is that the researcher must address the more fundamental issue of keyword bias when setting search parameters, or there is a high likelihood that important conversations will be missed. Minimising this risk means involving people with domain expertise.

“The widespread adoption of social media by consumers can only accelerate its adoption by the market research community in the coming years”

While the web is an incredible repository of information, there are topics and issues with inadequate coverage. We have found instances where clients’ concerns are either not being discussed, or are being discussed by groups that are not the primary interest of the client. While this may be relevant insight, it often doesn’t address the client’s business issues. In such instances, clients may want to consider strategies for generating conversations, such as building moderated communities or using other overt online interventions to stimulate discussion.

Representativeness – who’s out there?
The ability to use social media monitoring as a proxy for traditional market research is at the root of much of the criticism from the guardians of tradition. As indicated earlier, it is unlikely that social media will entirely replace the survey institution. However, criticism on the grounds of representativeness is somewhat disingenuous. While one could argue that social media monitoring is not hobbled by the artificiality of the traditional survey, a seasoned methodologist would no doubt point out that those who contribute content are, in all likelihood, not representative.

The criticism is similar to that levelled at online panels and web-based surveying ten years ago. The response is similar too: that social media can become a reliable market proxy through the use of weighting schemes. As was done with online panel surveys, developing appropriate weights involved repeated calibration against traditional survey results.

In regard to weighting, social media has another advantage. In this online environment data can be readily amassed moving forward and backward in time. Overlaying ongoing market events and business measures on social media time series data allows you to draw effective insights into the relationship of business outcomes and the rise and fall of conversations. The ease of developing a longitudinal approach even offers a considerable advantage over the traditional tracking study, precisely because social media monitoring is more adaptable to the changing discourse of the market

The future
Market researchers beware – social media monitoring is not going away. The proliferation of tools and services means the savvy adopter must critically evaluate everything that’s available. There is a wide range both in terms of capability and price, so understanding the relative merits of each solution is critical. Vendors pushing a black box should generate suspicion – the tools and capabilities are sufficiently advanced that anyone claiming to have a ‘secret sauce’ is likely hiding flaws and deficits.

The widespread adoption of social media by consumers can only accelerate its adoption by the market research community in the coming years. We are quickly moving past the hype, and most practitioners will soon be integrating social media monitoring into their research agendas.


13 years ago

Great and insightful article that covers all angles of social media monitoring. Arguments are unbiased and well-balanced. With regarfs to the acceleration in adoption rate of social media as a research tool, I could not agree more. The treasure trove of data online is just sitting there waiting to be mined for insights to provide better products and greater services. Ashley Social Media Consultant Brandtology

Like Report

13 years ago

Great article Scott! Your comment that many market research folks look at social media monitoring the same way people did at online research in 1999 is totally accurate. The obvious benefit is its unstructuredness; to learn what people are talking about rather than make answering specific questions that a brand / company wants to know (populace / consumer driven rather than company driven). The dangers as you mention are low end solutions that don't allow for much refinement, analysis, categorization or sentiment assignment, that could allow one to draw inaccurate conclusions. This is an area that will continue to grow, expand and refine itself over time. While the economy continues to be a challenge, people need to remember the old adage, "you get what you pay for." A $2k out of the box 'let's scrape the web and put it in some cool graphs' is a very different animal than custom applications that use a variety of tools and human analysis to arrive at the gems within social media monitoring!

Like Report

13 years ago

Lets face up to facts here, there are so many social media monitoring dashboards that are basically useless! the data they capture is flawed, and as for the comment by Brandtology, unbaised and well - balanced is NOT social media, it is far from it. people need to stop thinking social media dashboards are the answer.

Like Report