This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Find out more here

FEATURE10 June 2009

Tracking online word-of-mouth: The people vs machines debate

Features

Who’s best at sifting through online chatter to find the insights that businesses need? People or computers? Mark Westaby and Mike Daniels go head to head.

Mike Daniels, director of media analysis firm Report International, continues to swear by human analysis even when the content being examined is digital. Metrica founder Mark Westaby used to feel the same, but has come round to automated analysis, and is now one of its most vocal advocates. His new firm Spectrum has just launched its first text mining and sentiment analysis product. Research asked the pair of them to bat the issue back and forth by email.

From:  Mark Westaby
To:       Mike Daniels
Date:   19/5/09 15:22

Dear Mike

The internet is revolutionising the way people air their views. The result is a vast repository of comment and opinion, which has a much more powerful influence on consumers than advertising, marketing and even traditional editorial media coverage.

By monitoring and evaluating the sentiment of online media, including consumer-generated comments, organisations can gain significant advantage. Failing to do so can place them at a dangerous disadvantage. Negative impressions are exacerbated by the highly connected nature of the web, and in a world where sentiment can change and be transmitted to millions at the click of a mouse, evaluation must be virtually instantaneous.

The only way this can be achieved is through highly sophisticated automated systems, as human analysis simply cannot come close to the levels of consistent accuracy or response times required. Fortunately a new generation of technology permits consistently more accurate and cost-effective analysis of sentiment across online as well as traditional media. Critically, this allows monitoring and analysis of sentiment in real time, giving companies the intelligence they require to stay abreast of trends in market perception and the factors driving their reputation.

Best regards
Mark

From:  Mike Daniels
To:       Mark Westaby
Date:   25/5/09 19:24

Dear Mark

No one would dispute that the internet has had a profound and irreversible impact on consumers. Digital conversations are taking place in such large quantities that it is all too easy to believe that only automated tools can help us analyse the dynamics of this new word-of-mouth phenomenon.

The overwhelming majority of blogs and social media sites have an audience of two: the author and his mother

But there’s an unstated assumption behind the technology promise: that it is necessary to analyse all or a very large percentage of these conversations in case we miss something. Given that the overwhelming majority of blogs and social media sites have an audience of two (the author and his mother), it’s hard to imagine there is much real influence being exerted.

Even if we did want to track every single conversation, your assertion that automated analysis can yield accurate and consistent measures of sentiment flies in the face of research we conducted recently among a global sample of developers, practitioners, academics and users of these tools. We found no system capable of delivering reasonable accuracy levels around sentiment – certainly nowhere near the levels needed for making business decisions.

We have found an enduring demand for human-based measurement programmes – humans can discriminate irony and sarcasm, they can interpret rules, not just follow them, and they are flexible in dealing with new topics and issues… certainly not computers’ strong points.

Cheers
Mike

From:  Mark Westaby
To:       Mike Daniels
Date:   27/5/09 13:24

Dear Mike

Your argument about the blogger with the audience of two misses the point. The internet is a highly connected, non-random network, which means that even bloggers with tiny audiences are just a single click from having huge influence. This is how more and more crises are starting, with what at first appears to be an insignificant issue on a minor blog captured by, say, a journalist using a standard search engine, and that’s why it is necessary to analyse as much internet content as possible, quickly, so that problems can be identified early and nipped in the bud.

Best regards
Mark

From:  Mike Daniels
To:       Mark Westaby
Date:   28/5/09 17:46

Dear Mark

Your response illustrates perfectly how proponents of automated analysis always come back to speed as their most significant defining benefit. In crisis situations communicators need tools to help them determine the most appropriate tone and content for their response, as well as identifying where the critical pressure points are and where intervention will be most effective.ommunicators can ill afford to chase down false positives. Every single communicator I speak to about this issue, without exception, demands speed plus direction. Direction about the rate of growth or decline of the crisis issue, and direction about how best to react, and where. Waiting an hour or so more than an automated analysis with its inherent inaccuracies is definitely a price sensible communicators are willing to pay.

Best regards
Mike

From:  Mark Westaby
To:       Mike Daniels
Date:   29/5/09 10:55

Dear Mike

It’s not proponents of automated analysis but changes in our increasingly connected 24/7 world that are determining the value of automated systems for crisis management. If proponents of human analysis really believe that crisis situations make up a minority of work, I suggest they talk to more of the communication professionals for whom every day for the past several months has involved some crisis or another.

You repeat the common criticism that automation is not as accurate as human measurement. This assumes that automated systems are designed to replicate human analysis, which ours are not, for very sound reasons. The brain is a superb piece of machinery for coping with the complexity of human survival, but is actually remarkably poor at the cognitive demands of data coding, which forms the basis of human analysis. Automated systems are far better at this. And as those who support human analysis always fail to point out, analysing irony and sarcasm is one thing, but interpreting and coding them consistently is quite another.

Best regards
Mark

From:  Mike Daniels
To:       Mark Westaby
Date:   31/5/09 23:12

Dear Mark

Let me give an example of how easily false positives are generated and how damaging they can be. A client of ours in the technology sector was using an automated tool from another provider to measure sentiment in traditional and online media. One of their goals was to have their brand positively associated with environmental responsibility and green issues, so a great deal of effort was put into building a complex semantic model to measure this. Since the client had carried out no activities around the green brand message, you can imagine the consternation when it showed up strongly in the results feed. It turned out after a fair amount of digging (which, surprise surprise, had to be done by humans) that the word green had been used, correctly, in relation to a green coloured product that was produced with the aid of our client’s equipment. No connection directly to the client’s own products, and certainly no connection with anything to do with the environment. The client received a very rapid – but entirely false – reading of their brand’s media profile.

Of course humans don’t get everything right first time either, but in our experience humans make relatively few errors of judgement, especially in sentiment. Most automated solution providers that I know use human analysts to check the output of their tools, and in some cases to code at least some of the coverage – not the greatest vote of confidence in their automated outputs.

Best regards
Mike

From:  Mark Westaby
To:       Mike Daniels
Date:   3/6/09 15:38

Dear Mike

Your ‘green/environment’ example proves nothing about automated analysis except that there are some companies out there doing it badly – which can be said of human analysis too. There are a number of ways we can avoid false positives pretty much completely, while still delivering powerful sentiment analysis.

No amount of quality control or training can disguise the fact that humans are poor at consistently coding large volumes of complex data

It’s actually well established that today’s automated systems can achieve 80% accuracy against humans. But, and it’s a very big but, this assumes an automated system wants to be compared with humans. No amount of quality control or training can disguise the fact that humans are poor at consistently coding large volumes of complex data.

So should an organisation use an automated system for, say, monthly analysis of traditional press cuttings? Probably not, but should a company use human analysis for daily online news and blogs? Probably not. There is a place for both in today’s world. Proponents of human analysis would do well to accept that automated systems, properly used, might actually be rather better than they’re prepared to admit.

Best regards
Mark

From:  Mike Daniels
To:       Mark Westaby
Date:   7/6/09 16:27

Dear Mark

It’s an unpalatable truth for any automated system, but the only genuinely independent determinant of the accuracy of sentiment is human analysis.

In the end, this debate is more about determining where the dividing line lies between automated and human analysis than making a choice of one over the other. I would argue that clients could do much more to guard against quality concerns in media analysis (automated and human) by ensuring market researchers participate in the choice of solution and vendor. If there is one positive thing I would like to see come out of this debate, it is that MR professionals understand the need for their involvement in media analysis decision-making.

All of us in media analysis are still learning to understand the potential and, more importantly, the limitations of automated analysis tools. But as long as humans remain the final point of independent validation, computers can only ever remain a useful support and counting tool.

Cheers
Mike

15 Comments

8 years ago

It seems pretty clear that humans are the better option every time here. As Mike says, quick information that's incorrect is pretty worthless and in all likelihood downright dangerous. The common consensus in the US where data and text mining research is years ahead of the UK is that a blend of software and humans are what are needed. Is Mark providing this service or just backing a software alone system - which seems to be not the way the industry is moving - or needs to move.

Like Report

8 years ago

Great debate and I can see see the case for both approaches or a combination depending on the circumstances. Size is a key issue - if you are looking for topline indications across multiple products/brands/services, then automated is probably the way to go. If you're looking for greater depth of analysis - pos/neg/neutral or 1-10 scale assigned to at opinion level (on reference can contain opinions about many concepts, pos, neg or neutral) then human analysis or human moderated automatic analysis is the best bet. I agree with Mike's point that you don't need to look at everything.You also need to provide analysis of the impact of certain media - some of that automated (traffic rankings, link analysis) but some human (frequency of update, originality of content, how interactive is the media). Another issue/question - how is automated analysis developing to cope with langauges others than English?

Like Report

8 years ago

Great debate, and good idea to conduct it this way. I've been working in online PR and online opinion research for a number of years now, and I tend to side with Mike. I agree that computers are better at coding in a consistent fashion, but they're terrible at "decoding" what it is they're actually reading. Humans are better at it, hands down. Now, I'm surprised to read, in Mark's email that "It’s actually well established that today’s automated systems can achieve 80% accuracy against humans". I would love to see this piece of research. Is it the one where the bulk of results ends up in the "neutral" column by default? And what would the results look like over something like Twitter? and to your point, Twitter is definitely relevant for early-warning monitoring, but exceedingly difficult for a machine to analyze on the fly. One more point: as is often the case, this debates starts with the premise that the social web only speaks English. Nothing could be further from the truth, and big brands need multi-lingual capabilities. And outside of the English-only comfort zone, automated systems fare even worse.

Like Report

8 years ago

In response to 'anonymous' we agree that a combination of human/automated analysis is the best solution for traditional media evaluation, but that's not an area on which we're focusing. Indeed, we believe the demand for real-time analysis of online media will grow phenomenally over the next five years for which automated analysis will be the only practical solution. People are, however, missing two really fundamental points here, which are (a) that automation allows data to be analysed in proper real-time and (b) that this then enables very large volumes of data to be collected on a time series basis without any feedback 'contamination'. In fact this is remarkably straightforward and can be done with virtually no risk of false positives. As a result we can carry out truly amazing, extremely robust statistical analysis that would otherwise be impossible, which is revealing things that human analysis could not. An example, which we'll be reporting on shortly, is to track the impact of senior management spokespeople on their company's share price. Using time series analysis we can measure this against share price at extremely high levels of confidence (99% +), providing very valuable feedback for companies whose CEOs are very busy people and need strong evidence for the time they might be asked to give. Last but not least, something else automated analysis can do that human analysis cannot is to determine the strength of coverage from a search engine optimisation perspective. Everybody thinks keywords are critical for driving page ranking but these are actually very blunt measures and there are far, far more powerful algorithms that only automated text mining can reveal. Indeed, this is something we can do very easily. As a result we can tell companies how well their coverage is supporting their SEO strategy, which as search becomes the number one citerion is rapidly becoming as, if not more powerful than tone. It's relatively early days but I think you'll see automated analysis come into its own over the next few years.

Like Report

8 years ago

Most observers will agree, as do the authors it seems, that this is not a question of either/or. Much rather, it is about agreeing methods and standards that will deliver actual best-of-both-worlds results now and in the foreseeable future. In that sense, increased automation must be welcomed as it drives commoditisation by allowing the processing of vast amounts of data in ever-decreasing periods of time. So then this debate is really about where automation should end, and where human intervention should start. Non-English language is a key issue, and any long-term prediction must free itself from the 'cultural myopia' of English. Automated sentiment analysis in Mandarin, anyone? Semiotics, as a theory of communications, falls into syntactics, semantics and pragmatics. Machines are, and will be, better equipped to bulk-process information according to syntactic and semantic rules. However, meaning and understanding or, in a more targeted sense, impact, comes out of pragmatics. Machines will always only do what we tell them, so let's tell them as much as we can, and get them to compile and categorise ever more, 24/7. As a USP in media intelligence, however, we should continue to aim for the smartest humans, not the biggest or fastest machines. As with so many aspects of life, size is not everything.

Like Report

8 years ago

It seems to me that automated systems enable you to look back at what you did and what happened, but they don't function as well to inform what you should do going forward. One of the fundamental challenges I see with automated systems is they only look for what you tell them to look for. One of the primary purposes of media monitoring ought to be identifying new issues as they arise but before they are serious challenges or opportunities. Getting this glimpse of potential futures enables communications practitioners, and even businesses as a whole, to act strategically. Communications can manage media relations and businesses can manage products, services and policies against emerging issues. So, it seems to me, human monitoring gives an organization a better foundation for competitive advantage.

Like Report

8 years ago

Mike and Mark - thanks for a stimulating discussion. Mike said: "All of us in media analysis are still learning to understand the potential and, more importantly, the limitations of automated analysis tools. But as long as humans remain the final point of independent validation, computers can only ever remain a useful support and counting tool." This matches my personal experience in these matters. The automated system can attain a relatively high level of accuracy only if it is capable of "learning" from adjustments in sentiment. I started out with relatively poor accuracy that improved gradually, thanks to tweaks from the supplier and from several of us, double checking sentiment. This was an onerous and difficult task in a rapidly changing media environment of significant volume. The other comment I would make is that the perception of accuracy -- and the trust in it -- is nearly as important as the actual sentiment data. The summary sentiment is a "blunt instrument" and needs some kind of appropriate scale that accurately describes the myriad gradations of sentiment. For now, I'll tip my cap to the learned Mr. Daniels -- combination offers the best potential analysis at this point.

Like Report

8 years ago

I’ll address specific issues in a moment but let me first make a fundamental point regarding automated analysis, which so many people seem to be missing. Automated analysis should not be viewed as a replacement for human analysis. Rather, it is a different method that is opening up entirely new and tremendously exciting ways of analysing data. The analogy I like to use relates to the film industry when ‘talkies’ first came along. When this first happened producers just started to film plays and stage shows. In other words they didn’t understand or appreciate that talkies enabled film to be used in a completely different way, which would allow it to become a new medium in its own right. Likewise, I’ve found that people simply look upon automated analysis as an alternative way of carrying out analysis that would otherwise be done by humans. Yes, there are some applications where that is the case but of far greater importance are new areas, which automated analysis is opening up. In particular is the ability to generate large volumes of time series data, which allows very robust statistical analysis to be conducted that would be impossible with human analysis. The language debate is an interesting one but in our experience the overwhelming volume of online media coverage, at least in business terms, is in English. Indeed, an aggregated search will reveal at least 95% of domains being .com followed by .co.uk. So much so, in fact, that we haven’t carried out analysis in any other language for a long time (we can currently handle Spanish, French and Italian as well as English; and remember that Spanish is one of the world's major languages). Yes, languages such as Mandarin present particular problems (not least in terms of characters as well as structure) but necessity is the mother of invention so don’t be too surprised if this changes rapidly if China’s online presence overcomes the political issues currently faced re censorship, etc. Re the use of automation to conduct ‘open’ analysis there are a number of tools developing rapidly that address this very issue and the time when computers can be used to analyse “what’s there” rather than “what we tell them” isn’t far away. Indeed, it’s already possible though not quite yet on the scale required for commercial applications. As for the question of looking forward, this is something for which automated analysis is ideally suited – far more than human analysis, in fact. Neither machine nor human can predict the future but one major benefit of automated analysis is the volume and granularity of time series data that it generates, which enable very sophisticated predictive models to be developed. Of course, humans still need to interpret the results of these models but they provide a very robust basis for forward-looking decision-making that, again, is pretty well impossible with human analysis. Last but not least the question of competitive advantage where, again, automated analysis offers huge benefits. The reason is that automated systems can analyse a hundred companies just as easily and only a few seconds longer than it takes to analyse one. The breadth and depth of competitive information this creates is huge and much, much greater than is practical – or cost-effective – using human analysis.

Like Report

8 years ago

This is a great debate. It's something we think about quite a lot at Networked Insights. Our R&D team has some thoughts on our blog (http://bit.ly/gRb8T) that I think you'll find relevant. Thanks again for the great debate, Alex Fortney

Like Report

8 years ago

Let me comment the subject from the engineering point of view. It seems to follow from the great discussion above that there are three main requirements as to the monitoring online media: real-time quickness, accuracy and understanding the meaning of the discourse monitored. As I see, we all agree, that the third needs specific human abilities and involvement in the monitoring process. So, the problem is how to construct human-operated Monitor accurate enough and fast enough. What does it mean "accurate enough"? What level of accuracy we need? Some experts seem to overestimate the weight of accuracy, and compete to attain it as close as possible to 100%. However, accuracy is costly. It costs both work and time. 100% accuracy is nonsense from the economic and technical point of view. Problem of optimum accuracy is similar to the old problem of quality assurance in manufacturing sector. Industrial statisticians made great progress when they replaced 100% quality control with the Statistical Process Control (SPC), based on the study of only the small fraction of pieces (samples). Some experts of sentiment analysis of online media texts urge to collect 100% of utterances of positive or negative opinions in the texts examined (sorry for simplification) in order to attain the highest possible accuracy. They apply strong computer applications to do that and attain the accuracy of 70-90%. This is not necessary. It is not so hard to attain 97% accuracy of the sentiment's changes measurement (on 0,95 confidence level) using SPC approach, provided that the results of sophisticated discourse analysis (human work!) are implemented in the Monitor. In this way the Monitor can be built as human-operated and computer-aided tool and process for real-time monitoring.

Like Report

Results 1 to 10 of 15