Twitter doesn’t have the answer

What do people think about autonomous cars? Would they consider buying one, and what levels of autonomous driving are they comfortable with? What would be their concerns, and what do they get excited about? We are told that social media can provide answers to such questions by tapping into the personal views people share with their social media peers and the wider world. And I agree – as long as we do not use a too narrow definition of social media.

All the common social media monitoring platforms claim that they cover not just Twitter, YouTube, Facebook and Instagram, but also trillions of other websites such as blogs, forums, mainstream news, comments and reviews.  

However, in reality, whenever you create a query on any of these platforms, the overwhelming majority of content they return comes from Twitter. Sales reps for social media platforms tell me this huge share is a true representation of Twitter’s importance in terms of the amount of its consumer generated content. Do you believe this?

I don’t. I think the true reason is that Twitter (and Facebook and Instagram) provide a convenient, fire-hosed access to millions and billions of tweets and posts, which are much easier to retrieve, ingest and process than the non-normalised, un-structured and unwieldy content of millions of different websites, forums and blogs.  

Product reviews, for instance, are poorly represented on common social media monitoring platforms. You will find some reviews among your query results, but these (supposedly non-random) small samples cannot provide any deep insights about how consumers feel about a product or service, how they interact with it, or what they would like to see improved.  

I imagine the reason for this poor representation of consumer reviews lies in the fact there are numerous relevant websites depending on product category, each of which come with their particular challenges when trying to extract the plain text of reviews (such as different ways of handling pagination, page structure, Ajax calls, etc.). But I am just guessing.

Whatever the reason, this skewness towards mainstream, social mass-media channels drastically reduces our ability to extract the really valuable insights. We all know the stories of how Twitter data has been used for real product innovation, based on for instance people tweeting about their own Marmite-Mango smoothie creations and similar things. And I don’t want to rubbish these stories, they are great examples of the creative use of the data that is easily available.  

But when a systematic analysis of the strength and weakness of a product and its competitors is required, Twitter will not be able to provide the answer. This is partly because tweets are limited to just 140 characters, which leaves little space to provide elaborate opinions. But it is also because Twitter is simply not used for this kind of experience sharing.

Twitter is a vanity fair (we are all guilty, I don’t exclude myself), a stage for self-promotion, sometimes a channel for classified ads, and a (fake) news channel. But it is rarely helpful in understanding what is wrong with a product, and how to improve it.

Social media to me is everywhere where people generate and share content. Our job as researchers is to identify the relevant sources that can provide the insights that are required to help our clients, rather than to just blindly use the easily accessible ones that we are offered by social media monitoring platforms.  

And once we identified the sources, we can use tools like Python, R and JavaScript to build custom data extraction solutions – I don’t like the term web scraping, but that’s what I am talking about.  Such a DIY approach to social media analysis ensures that we gather the relevant content, in sufficient volume, with control over the sample.  This way we can extract real insights that go far beyond some brand buzz, sentiment and Marmite-Mango smoothies.

Frank Hedler is director advanced analytics at Simpson Carpenter