This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Find out more here

FEATURE16 September 2013

Social dominance

Features

Can social media analysis ever be a replacement for traditional market research? Gareth Price and Mark Westaby go head-to-head.

Gareth Price (above left) was the author of the first piece. He’s a researcher working in the brand insight division of Precise, and is responsible for developing the company’s social media research offer. His article was based around the findings of a Pew study which showed that traditional surveys and social media sentiment often diverge when it comes to measuring things like voting intention. Price’s argument, then, was that researchers need to accept that social media is different from traditional research and can never directly replace it.

Then there was Mark Westaby, the founder of SpectrumInsight. He sent us a blog post based on some internal research that found that Twitter-based ‘social’ insight correlates closely with traditional market research surveys regardless of demographics. Mark’s argument was that this finding opens the way for Twitter to be used as a legitimate market research tool over periods for which traditional market research would be prohibitively expensive, as well as enabling insights to be captured that would otherwise be missed.

We asked them to go head-to-head to debate the issues and answer the question once and for all: Can social media analysis ever be a replacement for traditional market research?

GP: Can it complement traditional methodologies? Absolutely. Replace? In some circumstances it might enable us to answer a question that’s otherwise difficult to answer. Will we ever be in a position where we no longer need to ask questions of respondents and collect all information passively? I don’t believe so.

At present, the people using social media don’t make up a representative sample of the population, so it cannot replicate a quantitative tracking survey, for example. It can help explain why opinions towards a brand may have changed. However, if we need to quantify the size of an issue, social media research can only do that in a specific context and can’t tell us how this is reflected among particular demographics and the population at large.

The unstructured nature of the data means we can’t always make it do what we want it to do either. Because of this, you have to think about it in more of a qualitative way. Even if this is at scale, if you’re advocating that it replaces all forms of research, you’re effectively making the argument for no longer doing quantitative research. Social media research informs traditional methodologies. It’s a holistic process.

MW: In many circumstances, social media analysis, which I’ll call ‘social insight’, isn’t just a replacement but is actually vastly superior to traditional market research for reasons that will become obvious.

Your opening salvo, Gareth, throws up that old favourite of the ‘representative sample’, which is how every traditional market researcher tries to attack social insight. However, what exactly is a ‘representative sample’ and what are such samples really representative of? The answer certainly isn’t ‘the real world’ because real people don’t behave in the nice ‘representative sample’ way that traditional market researchers love so much.

On any single day the ‘representative sample’ of people who use Tesco will not be the same as on the next day, or the day after; and that goes not only for every other supermarket but also for every single brand the supermarkets sell.

Furthermore, a ‘representative sample’ in itself is going to provide very limited insight. Knowing what the population might do is only going to tell me about Mr and Mrs Average. But it’s Mr and Mrs Not-Average that I really want to understand. But as soon as we drill down into our ‘representative sample’ to try to find out about Mr and Mrs Not-Average, our statistically significant data becomes not very statistically significant.

This is the fundamental problem with traditional market research: it’s limited to the samples it can collect. And the problem with samples is that their statistical significance is limited to the population they represent which, as we’ve seen, aren’t really very representative of what actually happens in the real world.

Social insight provides an ideal way of addressing all of the the above problems. For starters, let’s take the Tesco example used earlier. We track tweets from millions of unique supermarkets users, who in fact represent between 1 in 10 and 1 in 20 of all consumers who ever use a supermarket. With these numbers we’re not just tracking a sample but the population itself.

Furthermore we’re doing so on a 24/7 basis, which means we can track what consumers are thinking and doing literally by the hour.

“A ‘representative sample’ in itself
is going to provide very limited insight.
Knowing what the population might do
is only going to tell me about
Mr and Mrs Average. But it’s
Mr and Mrs Not-Average that I really
want to understand”

GP: I’m confused on a couple of points, Mark. You seem to be arguing that extracting data from a representative sample of the population no longer matters. In certain instances that may be the case but in polling, for example, it’s vitally important. The main reason Nate Silver has been so successful in predicting elections is that he’s able to aggregate lots of polling data that’s representative at the local and national level.

I think you’re wrong to say that you can’t drill down into a representative sample either. As long as you achieve results above a statistically significant level, you can talk about and investigate the implications in more detail (e.g. understanding attitudes by demographic – ABC1s v C2DEs, women vs. men, etc.).

If we’re talking about the fact there isn’t a ‘Mr and Mrs Average’ anymore, and everyone is a sample of one, then in-depth qualitative research is the only way to understand their behaviour. This can take the form of ethnography, observing people in store, using eye-tracking or even running focus groups. If you look at an agency like BrainJuicer they use all sorts of innovative ways of understanding a truer-to-life model of human behaviour. This can also take the form of social media analysis but at a very granular level.

I think we’re talking about quite a hypothetical level of social media use too. To suggest that lots of people are using these networks to convey every aspect of their shopping experience is either naïve or wilfully ignorant.

Social media research can help us answer some of the questions some of the time. Understanding the results it delivers means understanding the context in which people share experiences. It also means understanding in which context it works and in which context we need to turn to a different methodology.

MW: Regarding your comment about getting away from Mr and Mrs Average, I’m far from talking about a “sample of one”. Indeed, my point is entirely the opposite: social insight enables us to understand how Mr and Mrs Not-Average behave because it gives us access to lots of Mr and Mrs Not-Averages when – as you rightly point out – the only way traditional market research can hope to find out about them is through qualitative research.

What I’m arguing is that the importance of extracting data from a representative sample of the population – by which I take it you mean the UK population – has declined enormously over recent years to the extent that it is now of relatively minor value. The reason is that brands must now respond to a world where consumers have greater freedom of choice and the ability to make purchases on a 24/7 real-time basis, as a result of which understanding how actual consumers are behaving in real-time and on a continuous basis is of much greater value than having a snapshot of the UK population. The one major exception to this, I agree, is political polling, but that’s hardly the basis on which to argue that social insight cannot do a better job than traditional market research in many instances.

On top of this is the problem that people say one thing when answering even the most beautifully crafted questionnaire and then behave in completely the opposite way. Market research surveys rely on the cognitive responses of people who participate in them when, of course, human behaviour is driven mainly by emotions. The result of this is that a consumer can answer a question completely honestly and yet act quite differently without even being aware they are doing so.

The beauty of social insight here is that we’re using people’s natural language, unprompted by a market research question, which is important because people cannot disguise the language they use in written form – and even when deliberately trying to do so, they rapidly revert. This means we can find out far more accurately what consumers really feel and believe than is possible in even the best focus group.

“Social media research can help us
answer some of the questions some
of the time. Understanding the results
it delivers means understanding the
context in which people share
experiences”

GP: I’d like to start by reiterating that I do think social insight can do a better job than traditional methodologies in some instances. However, I think it’s important to emphasise the fact that different methodologies work in different contexts and help you to answer different questions.

As to your point about surveys and cognitive versus emotional response: Are you really suggesting that people always truly think and act in exactly the same way they do when they subsequently share this behaviour on social media? And that they always share it?

Many of the limitations you draw attention to in collecting survey data also apply to social media research. The level of usage is incomplete and the ways in which networks are used are interchangeable. Post-rationalisation can apply to a very short as well as a very long time-frame.

I’ve written quite a lot about digital dualism and how people inaccurately draw a distinction between the ‘real’ and the ‘virtual’ world when, of course, there is only one world. However, behaviour is always contextual and character limits, privacy settings and the type of people we’re friends with or follow all affect how we share information.

Short posts, for example, have the dual advantage of encouraging more people to join the network (as they make less demand on our time) and give the impression of a more active community.

Also, our social media use isn’t an expression of our subconscious desires and attitudes (something, incidentally, an Implicit Association Test can actually help you understand, for example).

I think we’re still at the stage where we need to prove the value of social media research. This means demonstrating business outcomes beyond replicating survey results and understanding it has a place, not the place in market research. That means accepting its limitations, as well as underlining its strengths.

MW: We are absolutely not interested in replicating market research, not least because with social insight we can do things that simply cannot be done in any other way. Nor do we believe social insight has the place in generating insight – but it can outperform traditional market research in crucial areas and to suggest that “we’re still at the stage where we need to prove the value of social media research” is to ignore, or misunderstand the very points that I’ve been making.

As for: “Our social media use isn’t an expression of our subconscious desires and attitudes” – I am not suggesting it is. The fact is, however, that consumers publicly express views and opinions on social media that are not cognitively biased by the intervention of market research questions. In addition, you mention that: “The level of usage is incomplete and the ways in which networks are used are interchangeable” and, of course, you are correct. But the use of social media is already so great that it represents every level of society, ranging from socio-demographic categorisation to people of all ages, including many over 65.

In summary, we are not saying that it’s about one methodology trumping all others; and nor are we saying that social insight is a panacea. What we are saying is that social insight can generate new insight that traditional market research will never be able to determine, while in many cases outperforming it.

GP: I also believe that social media research can generate insights that traditional market research would struggle to determine. However, it’s also important that we continue to be aware of the advantages of complementing what we do with other methodologies – which aren’t limited to the questionnaires and surveys you’ve focused on throughout. Only then can we select the best approach in each instance.

Gareth Price is knowledge leader at Precise. Mark Westaby is director at SpectrumInsight.

  • MRS is running a Social Media Research conference on 26 September in London with speakers from Mumsnet, Unlilever, and YouGov. Click here for details.

19 Comments

6 years ago

"At present, the people using social media don’t make up a representative sample of the population" This. Despite all other points, it boils down to which people are on social media, and what they're saying. At present, Twitter is still for a behaviourally different subgroup of people from the gen pop - they're tech-savvy, used to telling the world about themselves, and comfortable with all the implications of privacy (in a conventional, rather than cookie, sense) and exhibitionism that entails. In addition, social media analysis can get us to "what are people saying"; it still falls significantly short in "...and why are they saying it?". (even ignoring the hurdles still remaining in consistently and reliably 'measuring' social sentiment - as explained by Annie Petit)

Like Report

6 years ago

A clear win to Gareth. Daylight second.

Like Report

6 years ago

The comment about one-in-ten not being a sample, but being the whole population made me want to scream, but I blogged about it instead - see http://newmr.org/lets-protect-the-basic-truths-of-market-research-methodology/ New tools can be great, some will replace old tools, some will add to the tool chest, but few if any change the rules of evaluating whether something works properly. IMHO

Like Report

6 years ago

Sorry NickD, but you're wrong. First, it's simply incorrect to say that "Twitter is still for a behaviourally different subgroup of people from the gen pop". We can analyse users by age, location, interests, whether they're mums, dads, grandparents, students, etc, and they fall across all types of society. It's true to say there are fewer older people but even this is changing pretty rapidly. Second, Twitter is ideal for understanding "... and why are they saying it". Twitter reveals the consumer's agenda whereas traditional market research forces the researcher's agenda onto consumers. Twitter also reveals the consumer's true views and opinions without the (significant) cognitive influence of traditional market research. As to reliably 'measuring' social sentiment the vast majority of this consists of negative and positive analysis, which tells you virtually nothing regardless of its accuracy. What does provide real insight is the analysis of emotion, for which social media -- and Twitter in particular -- is ideal. Indeed, a major benefit of using Twitter this way is that it provides real-time analysis of emotion without having to tape electrodes to people's heads, show them facial expressions or any of the other methods that are fine in a lab-setting but not the real world in which consumers live.

Like Report

6 years ago

I think you’ve misunderstood the comment, Ray. Put simply, we can draw sub-samples from our massive sample that would otherwise be treated as outliers; and by analysing these comments we can determine things that traditional methods wouldn’t have a cat-in-hell’s chance of finding. How do we know this works? Because we can track our findings against completely independent data to verify and validate it, which we do rigorously and robustly. As a result we’re able to help brands understand things that have previously been complete mysteries.

Like Report

6 years ago

Mark, you say, and I quote, "We track tweets from millions of unique supermarkets users, who in fact represent between 1 in 10 and 1 in 20 of all consumers who ever use a supermarket. With these numbers we’re not just tracking a sample but the population itself." - Sorry but that is nonsense The ability to draw sub-samples that match key demographics of the population is useful, but only on the characteristics mapped, not on the attitude, belief, and behaviour differences between Tweeters and the 90%. Yes, a larger sample enables you to find more rare types, which is great, but is still a sample. I am a fan of social media research, I do it as part of my core work, and I find much of what you are saying a untenable.

Like Report

6 years ago

I'm sorry, Ray, but you're still missing the point. We don't mean sub-samples of the UK population but sub-samples of the supermarket's consumers, ie the supermarket's population (or whatever it is we're looking at). It doesn't matter a jot whether this matches the "key demographics" as you put it, or not. In most instances it won't -- I simply mention "key demographics" to explain that Twitter users are now completely ordinary folk and not this mythical race with which people associate them. Getting back to the supermarkets, what they need to know is what THEIR consumers think and how their consumers behave regardless of whether this matches the "UK population" or not (it rarely does). Believe me, if you're analysing between 1 in 10 and 1 in 20 of a supermarket's whole user-base you can find tremendous detail that would otherwise be impossible to see.

Like Report

6 years ago

I find the whole article a bit, pointless. Clearly Gareth accepts that this 'tool' or 'methodology' has a role, and that's that really. I am not sure how it goes any further. There have been ten dozen methodologies that claim to be superior which a good researcher will simply take on board as another of their tools and line it up alongside others that have appeared. I am not sure therefore what the argument is? If Mark has validated his method then fine (the argument would be more impactful if that was shown). I would however like to challenge a statement made in the comments above, that Twitter reveals someones true views and opinions. I find this an interesting statement to make. How many celebrity stories do we see where they have retracted a tweet because it wasn't really what they thought and that it was a spur of the moment thing? Yes that's what they thought at that second (probably under the influence of alcohol!) but is that what they really think? I imagine the case studies that show how great social media analysis is are fine, I would be just as interested to see where it doesn't and should not be used.

Like Report

6 years ago

>Sorry NickD, but you're wrong. First, it's simply incorrect to say that "Twitter is still for a behaviourally different subgroup of people from the gen pop". We can analyse users by age, location, interests, whether they're mums, dads, grandparents, students, etc, and they fall across all types of society. Don't think so. Twitter users *are* different from non-users, by definition - and the user base is still a sufficiently smalll proportion that that difference is possible. When it gets to Facebook levels of 80%+ penetration, it'll be less of an issue, but for now, there are differences. The other thing is, take a brand that has outlets in the UK, US, Canada and Australia - and then reliably separate out the social comments from the UK (or indeed Canada or Australia) from those in the US. Doing it by country of account registration is not reliable; language analytics is tricky (and also not always viable)

Like Report

6 years ago  |  1 like

Talking as someone with very little statistical knowledge, I can see how Twitter users are different from the majority of the general population who don't use Twtiter. However, doesn't that also mean that users who choose to partake in a survey also different from the majority of the general population who choose to not partake in surveys?

Like Report

Results 1 to 10 of 19