FEATURE3 September 2009

Is social media measurement meaningless?


Social media has caused a great deal of fuss in the worlds of marketing and research, but do the attempts to measure and make sense of all those friends, fans, followers, pokes, posts and retweets really amount to anything?

In a recent debate organised by the UK’s Internet Advertising Bureau, representatives of companies including TMW, Publicis Modem and Universal McCann discussed whether or not social media measurement is ‘meaningless’.

Here, Brad Little of Nielsen Online and Amy Kean of the IAB sum up the ‘yes’ and ‘no’ arguments.



Amy Kean
Senior marketing manager

Imagine you’re a busker. It’s wonderful to see people smiling away to your rendition of Killing Me Softly and giving you two thumbs up as they commute to work - you know you must be doing something right. But what matters to you at the day’s end is how much you made compared to yesterday, and whether you’re raking in more than your fellow buskers a little further down the street. The qualitative measures provided by the nods and smiles are definitely useful, but it’s those quantitative measures that pay the bills and make you successful.

Similarly, positive buzz and great reviews can make brands feel good about themselves – but surely this can only mean something when combined with the proof that our budgets are justified and a return on investment has been achieved. This is where it could be argued that social media measurement currently lacks meaning.

Let’s look at what we mean by measurement. Dictionary definitions boil it down to being about the ability to make comparisons with some kind of standard or sample. What does it take for that to ‘mean’ something? You could talk to me for 10 minutes in perfect Russian, but to me it would be meaningless unless I asked someone to translate.

“Case studies on social media are currently in short supply, and over the next couple of years more efforts should be made to take success stories straight to CMOs”

Which brings me to the first of four key reasons why social media measurement is meaningless: most senior management simply don’t understand it. There have been numerous studies that suggest ‘the board’ still haven’t fully got to grips with what social media is and how it works for brands. This, for many, is simply a waiting game. Case studies are currently in short supply, and over the next couple of years more efforts should be made to take success stories straight to CMOs to show exactly what can be done for brands. Part of the role of the IAB’s social media council is to educate the industry about the opportunities in this space.

Secondly, it remains hard to prove that social media contributes to the bottom line, unless of course you’re Dell. Few agencies are able to confidently make the claim that their social activity led to an increase in sales. In part this is because social media isn’t all about sales – it’s about developing long-term relationships and building brands over a period of time. But particularly when budgets are tight, profits and profits alone are what the bosses look for.

Reason three is that there’s no standardisation of metrics. Whilst the IAB in the US has taken steps to standardise definitions of social media metrics, we need to work together to help advertisers understand exactly how social media can meet your objectives in order for the measurement to actually make sense. If you asked five different social media practitioners what their most ‘successful’ activity was and why, you’d probably be presented with five very different brand stories, each with their own idea of what made it a success. Of course, this is part of what makes social media such a rich and dynamic area to work in, but it may also be detrimental to its widespread adoption. 

Lastly, there is currently little comparative benchmarking. Richard Pentin, strategist at TMW and esteemed member of the IAB social media council, recently gave a great example to prove this point: Uranus. The diameter of Uranus is 51,118 km. Its distance from the sun is 19 AU (astronomical units), its approximate density is 1.27 grams per cubic centimetre and it travels at an average speed of 15,290 miles per hour. These are impressive stats. Probably. But they’re meaningless for most people until you examine them within the context of the whole solar system. This is when we discover that Uranus is the third biggest planet and seventh from the sun - statistics that actually mean something.

If your social media activity generates 100,000 views on YouTube, gets you 40,000 Facebook friends and 20,000 positive blog mentions over the period of a month and your net promoter score rises, this may show incredible brand engagement. But what are we to measure it against for the campaign to be deemed a success? Once again, this is where case studies come in. We need agencies to share their stories in order to work out as an industry what the benchmarks should be for different sectors and demographics.

Therefore, as strong as the statement may sound, I would argue that today social media measurement is meaningless, but that this has the potential to change. We’re obsessed with numbers on campaign activity, and we need more convincing about the long-term effects of social media brand-building, enlisting advocates to drive sales along the way. Making it meaningful across the board is going to be a group exercise.



Brad Little
Director of industry solutions
Nielsen BuzzMetrics

The argument that it is too difficult for brands to measure the positive impact of social media campaigns misses the wood for the trees. We are in fact able to measure the amount, location and nature of conversations before, during and after a campaign.

Thinking of social media measurement as merely a way to measure campaigns only scratches the surface of this vast resource. Actually, another whole debate could be staged about the relevance of ‘campaign’-based activities in a social environment. The very mention of a campaign encourages us to think of when it will start and finish, whereas success in social media often depends on continuous conversations, which don’t have a clear beginning or end.

Many companies are of course measuring their activities in the medium itself, but are also using this source of information as a key research platform for their businesses. Once the focus widens from campaign measurement to include consumer research as well, the clear winner is the social media content and the insights it provides.

But how do we define social media? If we don’t know what it is that we want to measure, how can it be properly measured and evaluated? I tend to think of social media not in terms of the activities we can do in it, but in terms of the conversations and connections people are having within the medium.

The internet has become a multi-dimensional space based on connections between people. This has changed the marketing paradigm and the way brands interact with consumers. Today your brand is what people say it is. In today’s fragmented media, brands are realising that they must go out, find and interact with the consumer on their turf. We know that social media is where consumers are increasingly spending their time (about one in five minutes in the UK, says Nielsen – that’s more than email). Consumers are seeking accountability and transparency from the brands they relish the most.

“What social media data lacks in representative- ness it makes up for in the fact that it is unaided, organic and straight from real consumers”

When brands first learn about social media research, they often question the data’s representativeness, claiming that only tech-orientated people comment on these sites. Or they suggest that social media is just a place for people to whinge. Once the brands evaluate the research, however, they see that the content is becoming more representative by the day (the fastest-growing demographic of Facebook, for example, is 50+). Also, what the data lacks in representativeness it makes up for in the fact that it is unaided, organic and straight from real consumers. This is a research gem.

As for the types of people creating this content, which varies by the topic, more people are consuming the information than producing it, so that the impact on brands is even larger than imagined. Who hasn’t read an online review for that new gadget, perused Trip Advisor to assist with hotel selection or used internet research before purchasing a car? We have even seen this impact in our own online metered panel data. Finally, in the data our research teams analyse we tend to see more positive discussion than negative. Consumers are willing to become advocates and brand ambassadors, and their message is often received more positively and with more impact than other forms of marketing and communications.

When companies recognise a new form of insight, they realise that many different areas of their business are affected. The research and insights teams generally use the data as a new avenue to learn about customer needs and issues – and because the data is unaided, they often unearth the unexpected. Product development teams then address unmet customer needs or wants. Customer services can often cut down on call centre costs by building their own social presence, thereby turning this department from a cost centre to a retention, transparency and advocacy driver. Public relations, communications and marketing can all learn to communicate with their consumers in a way that will better resonate, and understand how their various activities affect discussion.

With all of these departments positively affected by social media measurement, it must be valuable.

This article draws on comments made by panel members at the IAB’s debate. Arguing ‘yes’ were Richard Pentin of TMW, Jill Lloyd of LBi and Steve Filler of Unruly Media. Arguing ‘no’ alongside Brad Little were Peter Kwong of Publicis Modem and Alastair Little of Universal McCann.


15 years ago

Good - if sometimes a bit well-worn - arguments on both sides. But I have to take issue with this statement: "Also, what the data lacks in representativeness it makes up for in the fact that it is unaided, organic and straight from real consumers." This is sleight of hand - these qualities simply have nothing to do with one another. Can't we have data that's representative AND "organic"? I think that constructing social network data as more 'natural' than survey data is dangerouns anyway - it encourages us to forget the contexts of online action. But that's another story!

Like Report

15 years ago

I Agreed with Brad’s points. In particular with regard to social media being about the conversations and connections and with Amy’s conclusions that more robust measurement is the way forward. My main criticism was towards Amy’s Busker analogy and how this fits in with buzz building. In the short term, a busker gets qualitative feedback on his performance which means nothing to him (financially) until other people who have seen him start telling other people about him. This way he will see an increase in his overall quantitative measure of cash when other people who have heard how good he is come to see him. Real world examples such as the Blair Witch Project built their buzz by relying on others to spread the word about their film well before it was released. District 9 took $90M in its first week. Prior to this a similar, but shorter term strategy on Twitter was used, “District 9” trended a number of times before the film was released. The format of twitter lends itself well to Consumer to Consumer communication and creating buzz. Regarding understanding the meaning, difficulties arise with translation only if one dimension is used. Someone speaking Russian only doesn’t make sense when isolated in one dimension. Other communications could allow you to guess the meaning. Such as the tone of the voice, or volume, timbre and intonation and physical communication. If the Russian was tapping their watch or got down on one knee clutching 12 red roses and an engagement ring, despite not being able to understand the words, the meaning could be understood.

Like Report

15 years ago

Nice to see this debated intelligently. The fragmentation of the social media analysis industry makes acheiving the standards for measurement quite difficult. As John Clarvis points out in the busker analogy, the importance of opinions expressed online is their impact on other people (C2C). These opinions may well not be representative but if they come up on a Google search when I'm about to buy a mobile device, book a hotel or open a bank account, they can definitely make an impact on my choice. Jon Moody www.asomo.net

Like Report

15 years ago

Comparative analysis is a good way to get at relevance. I think people overlook this competitor dimension and I've no idea why - at TCG we studied the launch of the first Google Android phone - it got 93,000 blog references. Seems a lot until you look across at competitors (iPhone got 1.2 million references in the same period) and you look at the trend (the Android phone got better buzz when nothing was happening!).

Like Report