Social dominance
Gareth Price (above left) was the author of the first piece. He’s a researcher working in the brand insight division of Precise, and is responsible for developing the company’s social media research offer. His article was based around the findings of a Pew study which showed that traditional surveys and social media sentiment often diverge when it comes to measuring things like voting intention. Price’s argument, then, was that researchers need to accept that social media is different from traditional research and can never directly replace it.
Then there was Mark Westaby, the founder of SpectrumInsight. He sent us a blog post based on some internal research that found that Twitter-based ‘social’ insight correlates closely with traditional market research surveys regardless of demographics. Mark’s argument was that this finding opens the way for Twitter to be used as a legitimate market research tool over periods for which traditional market research would be prohibitively expensive, as well as enabling insights to be captured that would otherwise be missed.
We asked them to go head-to-head to debate the issues and answer the question once and for all: Can social media analysis ever be a replacement for traditional market research? GP: Can it complement traditional methodologies? Absolutely. Replace? In some circumstances it might enable us to answer a question that’s otherwise difficult to answer. Will we ever be in a position where we no longer need to ask questions of respondents and collect all information passively? I don’t believe so. At present, the people using social media don’t make up a representative sample of the population, so it cannot replicate a quantitative tracking survey, for example. It can help explain why opinions towards a brand may have changed. However, if we need to quantify the size of an issue, social media research can only do that in a specific context and can’t tell us how this is reflected among particular demographics and the population at large. The unstructured nature of the data means we can’t always make it do what we want it to do either. Because of this, you have to think about it in more of a qualitative way. Even if this is at scale, if you’re advocating that it replaces all forms of research, you’re effectively making the argument for no longer doing quantitative research. Social media research informs traditional methodologies. It’s a holistic process. MW: In many circumstances, social media analysis, which I’ll call ‘social insight’, isn’t just a replacement but is actually vastly superior to traditional market research for reasons that will become obvious. Your opening salvo, Gareth, throws up that old favourite of the ‘representative sample’, which is how every traditional market researcher tries to attack social insight. However, what exactly is a ‘representative sample’ and what are such samples really representative of? The answer certainly isn’t ‘the real world’ because real people don’t behave in the nice ‘representative sample’ way that traditional market researchers love so much. On any single day the ‘representative sample’ of people who use Tesco will not be the same as on the next day, or the day after; and that goes not only for every other supermarket but also for every single brand the supermarkets sell. Furthermore, a ‘representative sample’ in itself is going to provide very limited insight. Knowing what the population might do is only going to tell me about Mr and Mrs Average. But it’s Mr and Mrs Not-Average that I really want to understand. But as soon as we drill down into our ‘representative sample’ to try to find out about Mr and Mrs Not-Average, our statistically significant data becomes not very statistically significant. This is the fundamental problem with traditional market research: it’s limited to the samples it can collect. And the problem with samples is that their statistical significance is limited to the population they represent which, as we’ve seen, aren’t really very representative of what actually happens in the real world. Social insight provides an ideal way of addressing all of the the above problems. For starters, let’s take the Tesco example used earlier. We track tweets from millions of unique supermarkets users, who in fact represent between 1 in 10 and 1 in 20 of all consumers who ever use a supermarket. With these numbers we’re not just tracking a sample but the population itself. Furthermore we’re doing so on a 24/7 basis, which means we can track what consumers are thinking and doing literally by the hour. GP: I’m confused on a couple of points, Mark. You seem to be arguing that extracting data from a representative sample of the population no longer matters. In certain instances that may be the case but in polling, for example, it’s vitally important. The main reason Nate Silver has been so successful in predicting elections is that he’s able to aggregate lots of polling data that’s representative at the local and national level. I think you’re wrong to say that you can’t drill down into a representative sample either. As long as you achieve results above a statistically significant level, you can talk about and investigate the implications in more detail (e.g. understanding attitudes by demographic – ABC1s v C2DEs, women vs. men, etc.). If we’re talking about the fact there isn’t a ‘Mr and Mrs Average’ anymore, and everyone is a sample of one, then in-depth qualitative research is the only way to understand their behaviour. This can take the form of ethnography, observing people in store, using eye-tracking or even running focus groups. If you look at an agency like BrainJuicer they use all sorts of innovative ways of understanding a truer-to-life model of human behaviour. This can also take the form of social media analysis but at a very granular level. I think we’re talking about quite a hypothetical level of social media use too. To suggest that lots of people are using these networks to convey every aspect of their shopping experience is either naïve or wilfully ignorant. Social media research can help us answer some of the questions some of the time. Understanding the results it delivers means understanding the context in which people share experiences. It also means understanding in which context it works and in which context we need to turn to a different methodology. MW: Regarding your comment about getting away from Mr and Mrs Average, I’m far from talking about a “sample of one”. Indeed, my point is entirely the opposite: social insight enables us to understand how Mr and Mrs Not-Average behave because it gives us access to lots of Mr and Mrs Not-Averages when – as you rightly point out – the only way traditional market research can hope to find out about them is through qualitative research. What I’m arguing is that the importance of extracting data from a representative sample of the population – by which I take it you mean the UK population – has declined enormously over recent years to the extent that it is now of relatively minor value. The reason is that brands must now respond to a world where consumers have greater freedom of choice and the ability to make purchases on a 24/7 real-time basis, as a result of which understanding how actual consumers are behaving in real-time and on a continuous basis is of much greater value than having a snapshot of the UK population. The one major exception to this, I agree, is political polling, but that’s hardly the basis on which to argue that social insight cannot do a better job than traditional market research in many instances. On top of this is the problem that people say one thing when answering even the most beautifully crafted questionnaire and then behave in completely the opposite way. Market research surveys rely on the cognitive responses of people who participate in them when, of course, human behaviour is driven mainly by emotions. The result of this is that a consumer can answer a question completely honestly and yet act quite differently without even being aware they are doing so. The beauty of social insight here is that we’re using people’s natural language, unprompted by a market research question, which is important because people cannot disguise the language they use in written form – and even when deliberately trying to do so, they rapidly revert. This means we can find out far more accurately what consumers really feel and believe than is possible in even the best focus group. GP: I’d like to start by reiterating that I do think social insight can do a better job than traditional methodologies in some instances. However, I think it’s important to emphasise the fact that different methodologies work in different contexts and help you to answer different questions. As to your point about surveys and cognitive versus emotional response: Are you really suggesting that people always truly think and act in exactly the same way they do when they subsequently share this behaviour on social media? And that they always share it? Many of the limitations you draw attention to in collecting survey data also apply to social media research. The level of usage is incomplete and the ways in which networks are used are interchangeable. Post-rationalisation can apply to a very short as well as a very long time-frame. I’ve written quite a lot about digital dualism and how people inaccurately draw a distinction between the ‘real’ and the ‘virtual’ world when, of course, there is only one world. However, behaviour is always contextual and character limits, privacy settings and the type of people we’re friends with or follow all affect how we share information. Short posts, for example, have the dual advantage of encouraging more people to join the network (as they make less demand on our time) and give the impression of a more active community. Also, our social media use isn’t an expression of our subconscious desires and attitudes (something, incidentally, an Implicit Association Test can actually help you understand, for example). I think we’re still at the stage where we need to prove the value of social media research. This means demonstrating business outcomes beyond replicating survey results and understanding it has a place, not the place in market research. That means accepting its limitations, as well as underlining its strengths. MW: We are absolutely not interested in replicating market research, not least because with social insight we can do things that simply cannot be done in any other way. Nor do we believe social insight has the place in generating insight – but it can outperform traditional market research in crucial areas and to suggest that “we’re still at the stage where we need to prove the value of social media research” is to ignore, or misunderstand the very points that I’ve been making. As for: “Our social media use isn’t an expression of our subconscious desires and attitudes” – I am not suggesting it is. The fact is, however, that consumers publicly express views and opinions on social media that are not cognitively biased by the intervention of market research questions. In addition, you mention that: “The level of usage is incomplete and the ways in which networks are used are interchangeable” and, of course, you are correct. But the use of social media is already so great that it represents every level of society, ranging from socio-demographic categorisation to people of all ages, including many over 65. In summary, we are not saying that it’s about one methodology trumping all others; and nor are we saying that social insight is a panacea. What we are saying is that social insight can generate new insight that traditional market research will never be able to determine, while in many cases outperforming it. GP: I also believe that social media research can generate insights that traditional market research would struggle to determine. However, it’s also important that we continue to be aware of the advantages of complementing what we do with other methodologies – which aren’t limited to the questionnaires and surveys you’ve focused on throughout. Only then can we select the best approach in each instance. Gareth Price is knowledge leader at Precise. Mark Westaby is director at SpectrumInsight.
“A ‘representative sample’ in itself
is going to provide very limited insight.
Knowing what the population might do
is only going to tell me about
Mr and Mrs Average. But it’s
Mr and Mrs Not-Average that I really
want to understand”“Social media research can help us
answer some of the questions some
of the time. Understanding the results
it delivers means understanding the
context in which people share
experiences”
We hope you enjoyed this article.
Research Live is published by MRS.
The Market Research Society (MRS) exists to promote and protect the research sector, showcasing how research delivers impact for businesses and government.
Members of MRS enjoy many benefits including tailoured policy guidance, discounts on training and conferences, and access to member-only content.
For example, there's an archive of winning case studies from over a decade of MRS Awards.
Find out more about the benefits of joining MRS here.
19 Comments
NickD
12 years ago
"At present, the people using social media don’t make up a representative sample of the population" This. Despite all other points, it boils down to which people are on social media, and what they're saying. At present, Twitter is still for a behaviourally different subgroup of people from the gen pop - they're tech-savvy, used to telling the world about themselves, and comfortable with all the implications of privacy (in a conventional, rather than cookie, sense) and exhibitionism that entails. In addition, social media analysis can get us to "what are people saying"; it still falls significantly short in "...and why are they saying it?". (even ignoring the hurdles still remaining in consistently and reliably 'measuring' social sentiment - as explained by Annie Petit)
Like Reply Report
John Grono
12 years ago
A clear win to Gareth. Daylight second.
Like Reply Report
Ray Poynter
12 years ago
The comment about one-in-ten not being a sample, but being the whole population made me want to scream, but I blogged about it instead - see http://newmr.org/lets-protect-the-basic-truths-of-market-research-methodology/ New tools can be great, some will replace old tools, some will add to the tool chest, but few if any change the rules of evaluating whether something works properly. IMHO
Like Reply Report
Mark Westaby
12 years ago
Sorry NickD, but you're wrong. First, it's simply incorrect to say that "Twitter is still for a behaviourally different subgroup of people from the gen pop". We can analyse users by age, location, interests, whether they're mums, dads, grandparents, students, etc, and they fall across all types of society. It's true to say there are fewer older people but even this is changing pretty rapidly. Second, Twitter is ideal for understanding "... and why are they saying it". Twitter reveals the consumer's agenda whereas traditional market research forces the researcher's agenda onto consumers. Twitter also reveals the consumer's true views and opinions without the (significant) cognitive influence of traditional market research. As to reliably 'measuring' social sentiment the vast majority of this consists of negative and positive analysis, which tells you virtually nothing regardless of its accuracy. What does provide real insight is the analysis of emotion, for which social media -- and Twitter in particular -- is ideal. Indeed, a major benefit of using Twitter this way is that it provides real-time analysis of emotion without having to tape electrodes to people's heads, show them facial expressions or any of the other methods that are fine in a lab-setting but not the real world in which consumers live.
Like Reply Report
Mark Westaby
12 years ago
I think you’ve misunderstood the comment, Ray. Put simply, we can draw sub-samples from our massive sample that would otherwise be treated as outliers; and by analysing these comments we can determine things that traditional methods wouldn’t have a cat-in-hell’s chance of finding. How do we know this works? Because we can track our findings against completely independent data to verify and validate it, which we do rigorously and robustly. As a result we’re able to help brands understand things that have previously been complete mysteries.
Like Reply Report
Ray Poynter
12 years ago
Mark, you say, and I quote, "We track tweets from millions of unique supermarkets users, who in fact represent between 1 in 10 and 1 in 20 of all consumers who ever use a supermarket. With these numbers we’re not just tracking a sample but the population itself." - Sorry but that is nonsense The ability to draw sub-samples that match key demographics of the population is useful, but only on the characteristics mapped, not on the attitude, belief, and behaviour differences between Tweeters and the 90%. Yes, a larger sample enables you to find more rare types, which is great, but is still a sample. I am a fan of social media research, I do it as part of my core work, and I find much of what you are saying a untenable.
Like Reply Report
Mark Westaby
12 years ago
I'm sorry, Ray, but you're still missing the point. We don't mean sub-samples of the UK population but sub-samples of the supermarket's consumers, ie the supermarket's population (or whatever it is we're looking at). It doesn't matter a jot whether this matches the "key demographics" as you put it, or not. In most instances it won't -- I simply mention "key demographics" to explain that Twitter users are now completely ordinary folk and not this mythical race with which people associate them. Getting back to the supermarkets, what they need to know is what THEIR consumers think and how their consumers behave regardless of whether this matches the "UK population" or not (it rarely does). Believe me, if you're analysing between 1 in 10 and 1 in 20 of a supermarket's whole user-base you can find tremendous detail that would otherwise be impossible to see.
Like Reply Report
Anon
12 years ago
I find the whole article a bit, pointless. Clearly Gareth accepts that this 'tool' or 'methodology' has a role, and that's that really. I am not sure how it goes any further. There have been ten dozen methodologies that claim to be superior which a good researcher will simply take on board as another of their tools and line it up alongside others that have appeared. I am not sure therefore what the argument is? If Mark has validated his method then fine (the argument would be more impactful if that was shown). I would however like to challenge a statement made in the comments above, that Twitter reveals someones true views and opinions. I find this an interesting statement to make. How many celebrity stories do we see where they have retracted a tweet because it wasn't really what they thought and that it was a spur of the moment thing? Yes that's what they thought at that second (probably under the influence of alcohol!) but is that what they really think? I imagine the case studies that show how great social media analysis is are fine, I would be just as interested to see where it doesn't and should not be used.
Like Reply Report
NickD
12 years ago
>Sorry NickD, but you're wrong. First, it's simply incorrect to say that "Twitter is still for a behaviourally different subgroup of people from the gen pop". We can analyse users by age, location, interests, whether they're mums, dads, grandparents, students, etc, and they fall across all types of society. Don't think so. Twitter users *are* different from non-users, by definition - and the user base is still a sufficiently smalll proportion that that difference is possible. When it gets to Facebook levels of 80%+ penetration, it'll be less of an issue, but for now, there are differences. The other thing is, take a brand that has outlets in the UK, US, Canada and Australia - and then reliably separate out the social comments from the UK (or indeed Canada or Australia) from those in the US. Doing it by country of account registration is not reliable; language analytics is tricky (and also not always viable)
Like Reply Report
Jen D
12 years ago | 1 like
Talking as someone with very little statistical knowledge, I can see how Twitter users are different from the majority of the general population who don't use Twtiter. However, doesn't that also mean that users who choose to partake in a survey also different from the majority of the general population who choose to not partake in surveys?
Like Reply Report
Mark Westaby
12 years ago
Sorry NickD, but you're STILL wrong. Using your argument I could say that people who answer telephone surveys are different from those that don't. I could even go so far as to say that the base of people who answer telephone surveys is also a sufficiently small proportion that the difference between them and the rest of the population is significant. How? Because we've carried our own study, which revealed that around 90% of the UK population decline to take part in telephone-based interviews. So by your reckoning telephone-based surveys are no longer appropriate. That is, of course, a ridiculous conclusion but it's what you're saying. Regarding locations we can and do very reliably sort out exactly where tweets come from and not by the method you mention, which is indeed very unreliable. We've double-checked our method, which is relatively easy given that people often tweet about where they are (Scunthorpe ASDA, for instance, or the BP garage in Gerrards Cross) and the match between this and the location we have for them is around 95%. We can determine whether tweets about Starbucks or McDonald's, for instance, are from the UK, USA or anywhere else. The same for any other brand you mention. We can also use this to determine where brands are gaining traction, for example we know that Irish Whiskey is gaining a lot of ground in Africa. No doubt you'll come back with some other reason to 'undermine' our work. That's fine, you carry on believing what you want and we'll continue based on the facts.
Like Reply Report
Anon
12 years ago
Just referring back to my comment which probably got missed in the other chatter, but how do you substantiate the statement that you made Mark that 'What people say on Social Media is a true view or opinion'. I have seen many a tweet by a friend or relative (in the short time I used it before getting bored) which clearly wasn't true and was an emotional vent or such but didn't actually replicate their true opinion. Also Ray Poynter raised some good comments around your claims to having access to a population rather than a sample, I would be interested in seeing your responses to these.
Like Reply Report
Ray Poynter
12 years ago
@Mark, you need to be more careful with details and MR practice. Yes, the response rate to telephone surveys is less than 100%, 5%-15% are typical. In addition not everybody can be reached by telephone. That is why market researchers refer to it as a sample and talk about total error, taking into account the sample frame and response error amongst other elements. No market researcher would describe the people in a telephone sample as anything other than a sample - but you called your 5% to 10% of the population you can track as the population - perhaps just re-draft that statement? Also, you cannot track where everybody tweets from. Tweets come in two varieties, either located by the position of the tweeter or by the location that the person registering says they are at - typically their home location. It is fine to say they the difference is rarely material, but it is not fine to say you can always locate the place of tweeting. Next week is ESOMAR Congress in Istanbul, I urge people to capture tweets and compare the locations, some will be Turkey, but many will be UK, USA etc – and the content of the tweets will usually indicate where they are.
Like Reply Report
Mark Westaby
12 years ago
OK, let me address these comments. Anon, you say that tweets represent an "emotional vent", which is the very point that I'm making and exactly why they are so powerful. What you refer to above as people's "true opinion" is their cognitive response, usually expressed some time after the event and typically when they've "had chance to calm down". The key point here is that emotional responses are far more powerful in terms of how we actually behave than cognitive ones, which is why people so frequently say one thing and then behave in a completely different manner. The reason is that the emotional parts of our brain have been around for a lot, lot longer than the cognitive ones because they helped our ancestors to avoid danger. That's why we often react, say, by doing an emergency stop when a child runs into the road before realising what's happened. Twitter enables us to take advantage of this by capturing the "emotional vent" as you put it, which can provide far more powerful insight into how consumers really react, say, to brands than the cognitive responses they typically give in research questionnaires. You can see how we used this to predict that Bob Diamond would resign from Barclays when everybody else was saying he would survive by going to this YouTube link: http://www.youtube.com/watch?v=ho9HOTdHih0 Re Ray's point, I've addressed the comment about population above, which is that I was referring not to the UK population but to the population of a supermarket's consumer base (we track more than 1 million individual users of supermarkets, 24 x 7 x 52). But I completely accept the point Ray's making and I should have been clearer! As for telephone interviews I was simply making the point in response to an earlier comment that people who are on Twitter are somehow "different". They might have been, even just a couple of years ago, but it's now a myth. Twitter has grown at a staggering pace and we now see every part of society -- and population -- represented. As for location, Ray, we're rarely interested in exactly where people are tweeting from. The point I was making was in response to somebody who claimed we couldn't differentiate between, say, people who tweet about McDonald's in the UK vs those who do so in the States. Although you're correct that we can't do this for very person who tweets, we are able to do so for those who give their location on their public profiles, which is a very significant number. This can range from "I live in Aberdeen" to "Based in Boston, MA" but the result is that we can filter these tweets and analyse them accordingly. Last but not least, I also explained above that we've rigorously tested this method and in 95% of cases the people who tweet about, say, about "a BP garage in Leicester" are located using our method in that city.
Like Reply Report
Anon
12 years ago
I appreciate the 'emotional vent' is something you can't capture in research generally, and social media is great for picking that up, but given that someone might vent on social media, given the clear light of day what do they actually do? Any typical product change for instance, you see customers venting about the change on social media about how they will never buy that product again, do all those people then stop buying the product? Is there a very distinct difference between immediate opinion and longer term behaviour? I guess thats why I raised the point of 'where it doesn't work' as an important part of the understanding of social media analysis.
Like Reply Report
Mark Westaby
12 years ago
There can be a very significant difference between emotional and cognitive responses in terms of their ability to predict behaviour, Anon. For instance, this is one reason why traditional customer satisfaction surveys are failing to spot problems early enough for brands to react to them. For example, we could see from our Twitter analysis that it was very likely Tesco was heading for trouble, so it was no surprise to us when they hit the problems they did; and about a year ago we warned that Morrisons' problems were much more fundamental than Dalton Philips, the CEO, was claiming. He puts their poor results down to a lack of convenience stores and online service but our analysis suggested Morrisons' existing stores and food range were just as much, if not more to blame. One year on and Morrisons continues to lose market share, which we believe will continue unless and until they get these basics sorted out. Regarding consumers saying things such as they'll "never buy a product again", first, this has to be seen in the context of all comments, in which case it might be a relatively small response. Second, while it's true to say consumers don't always say why they'll "never buy something again", it's also true that they frequently will; and understanding why can provide extremely valuable insight. Third, we analyse emotional response into seven emotional categories (based on Paul Ekman's work), which can provide in-depth insight into, say, whether people are disappointed (sad) about a new product or if they might be angry that it's replaced one they loved. We're not saying this should replace traditional research, but we absolutely believe -- in fact we know -- that using our type of insight can add a vast amount of insight that in many cases would otherwise be missed.
Like Reply Report
Anon
12 years ago
Thanks Mark, I think your last paragraph is how I phrased it in my first post, that this is another tool in the capability set of a good researcher. Which is why I was slightly exasperated that it was almost set up as a you vs them type article! I totally believe you have to add the emotion to the 'cold light of day' understanding.
Like Reply Report
Anon
12 years ago
I am no expert - but a simple observation: Starting your argument with 'social insight isn’t just a replacement but is actually vastly superior to traditional market research for reasons that will become obvious' ... and then ending your last comment with "We're not saying this should replace traditional research" seems to be a pretty major change in stance! The value of social insight risks being lost in the detail of this argument...
Like Reply Report
Mark Westaby
12 years ago
Anon, you have conveniently missed out three crucial words that I put in my opening statement, "In many circumstances...". So what I actually said is entirely consistent with the final comment. As for "detail of the argument", details ALWAYS matter.
Like Reply Report