Tracking online word-of-mouth: The people vs machines debate

It seems pretty clear that humans are the better option every time here. As Mike says, quick information that's incorrect is pretty worthless and in all likelihood downright dangerous. The common consensus in the US where data and text mining research is years ahead of the UK is that a blend of software and humans are what are needed. Is Mark providing this service or just backing a software alone system - which seems to be not the way the industry is moving - or needs to move.

Like Reply Report

Great debate and I can see see the case for both approaches or a combination depending on the circumstances. Size is a key issue - if you are looking for topline indications across multiple products/brands/services, then automated is probably the way to go. If you're looking for greater depth of analysis - pos/neg/neutral or 1-10 scale assigned to at opinion level (on reference can contain opinions about many concepts, pos, neg or neutral) then human analysis or human moderated automatic analysis is the best bet. I agree with Mike's point that you don't need to look at everything.You also need to provide analysis of the impact of certain media - some of that automated (traffic rankings, link analysis) but some human (frequency of update, originality of content, how interactive is the media). Another issue/question - how is automated analysis developing to cope with langauges others than English?

Like Reply Report

Great debate, and good idea to conduct it this way. I've been working in online PR and online opinion research for a number of years now, and I tend to side with Mike. I agree that computers are better at coding in a consistent fashion, but they're terrible at "decoding" what it is they're actually reading. Humans are better at it, hands down. Now, I'm surprised to read, in Mark's email that "It’s actually well established that today’s automated systems can achieve 80% accuracy against humans". I would love to see this piece of research. Is it the one where the bulk of results ends up in the "neutral" column by default? And what would the results look like over something like Twitter? and to your point, Twitter is definitely relevant for early-warning monitoring, but exceedingly difficult for a machine to analyze on the fly. One more point: as is often the case, this debates starts with the premise that the social web only speaks English. Nothing could be further from the truth, and big brands need multi-lingual capabilities. And outside of the English-only comfort zone, automated systems fare even worse.

Like Reply Report

In response to 'anonymous' we agree that a combination of human/automated analysis is the best solution for traditional media evaluation, but that's not an area on which we're focusing. Indeed, we believe the demand for real-time analysis of online media will grow phenomenally over the next five years for which automated analysis will be the only practical solution. People are, however, missing two really fundamental points here, which are (a) that automation allows data to be analysed in proper real-time and (b) that this then enables very large volumes of data to be collected on a time series basis without any feedback 'contamination'. In fact this is remarkably straightforward and can be done with virtually no risk of false positives. As a result we can carry out truly amazing, extremely robust statistical analysis that would otherwise be impossible, which is revealing things that human analysis could not. An example, which we'll be reporting on shortly, is to track the impact of senior management spokespeople on their company's share price. Using time series analysis we can measure this against share price at extremely high levels of confidence (99% +), providing very valuable feedback for companies whose CEOs are very busy people and need strong evidence for the time they might be asked to give. Last but not least, something else automated analysis can do that human analysis cannot is to determine the strength of coverage from a search engine optimisation perspective. Everybody thinks keywords are critical for driving page ranking but these are actually very blunt measures and there are far, far more powerful algorithms that only automated text mining can reveal. Indeed, this is something we can do very easily. As a result we can tell companies how well their coverage is supporting their SEO strategy, which as search becomes the number one citerion is rapidly becoming as, if not more powerful than tone. It's relatively early days but I think you'll see automated analysis come into its own over the next few years.

Like Reply Report

Most observers will agree, as do the authors it seems, that this is not a question of either/or. Much rather, it is about agreeing methods and standards that will deliver actual best-of-both-worlds results now and in the foreseeable future. In that sense, increased automation must be welcomed as it drives commoditisation by allowing the processing of vast amounts of data in ever-decreasing periods of time. So then this debate is really about where automation should end, and where human intervention should start. Non-English language is a key issue, and any long-term prediction must free itself from the 'cultural myopia' of English. Automated sentiment analysis in Mandarin, anyone? Semiotics, as a theory of communications, falls into syntactics, semantics and pragmatics. Machines are, and will be, better equipped to bulk-process information according to syntactic and semantic rules. However, meaning and understanding or, in a more targeted sense, impact, comes out of pragmatics. Machines will always only do what we tell them, so let's tell them as much as we can, and get them to compile and categorise ever more, 24/7. As a USP in media intelligence, however, we should continue to aim for the smartest humans, not the biggest or fastest machines. As with so many aspects of life, size is not everything.

Like Reply Report

It seems to me that automated systems enable you to look back at what you did and what happened, but they don't function as well to inform what you should do going forward. One of the fundamental challenges I see with automated systems is they only look for what you tell them to look for. One of the primary purposes of media monitoring ought to be identifying new issues as they arise but before they are serious challenges or opportunities. Getting this glimpse of potential futures enables communications practitioners, and even businesses as a whole, to act strategically. Communications can manage media relations and businesses can manage products, services and policies against emerging issues. So, it seems to me, human monitoring gives an organization a better foundation for competitive advantage.

Like Reply Report

Mike and Mark - thanks for a stimulating discussion. Mike said: "All of us in media analysis are still learning to understand the potential and, more importantly, the limitations of automated analysis tools. But as long as humans remain the final point of independent validation, computers can only ever remain a useful support and counting tool." This matches my personal experience in these matters. The automated system can attain a relatively high level of accuracy only if it is capable of "learning" from adjustments in sentiment. I started out with relatively poor accuracy that improved gradually, thanks to tweaks from the supplier and from several of us, double checking sentiment. This was an onerous and difficult task in a rapidly changing media environment of significant volume. The other comment I would make is that the perception of accuracy -- and the trust in it -- is nearly as important as the actual sentiment data. The summary sentiment is a "blunt instrument" and needs some kind of appropriate scale that accurately describes the myriad gradations of sentiment. For now, I'll tip my cap to the learned Mr. Daniels -- combination offers the best potential analysis at this point.

Like Reply Report

I’ll address specific issues in a moment but let me first make a fundamental point regarding automated analysis, which so many people seem to be missing. Automated analysis should not be viewed as a replacement for human analysis. Rather, it is a different method that is opening up entirely new and tremendously exciting ways of analysing data. The analogy I like to use relates to the film industry when ‘talkies’ first came along. When this first happened producers just started to film plays and stage shows. In other words they didn’t understand or appreciate that talkies enabled film to be used in a completely different way, which would allow it to become a new medium in its own right. Likewise, I’ve found that people simply look upon automated analysis as an alternative way of carrying out analysis that would otherwise be done by humans. Yes, there are some applications where that is the case but of far greater importance are new areas, which automated analysis is opening up. In particular is the ability to generate large volumes of time series data, which allows very robust statistical analysis to be conducted that would be impossible with human analysis. The language debate is an interesting one but in our experience the overwhelming volume of online media coverage, at least in business terms, is in English. Indeed, an aggregated search will reveal at least 95% of domains being .com followed by .co.uk. So much so, in fact, that we haven’t carried out analysis in any other language for a long time (we can currently handle Spanish, French and Italian as well as English; and remember that Spanish is one of the world's major languages). Yes, languages such as Mandarin present particular problems (not least in terms of characters as well as structure) but necessity is the mother of invention so don’t be too surprised if this changes rapidly if China’s online presence overcomes the political issues currently faced re censorship, etc. Re the use of automation to conduct ‘open’ analysis there are a number of tools developing rapidly that address this very issue and the time when computers can be used to analyse “what’s there” rather than “what we tell them” isn’t far away. Indeed, it’s already possible though not quite yet on the scale required for commercial applications. As for the question of looking forward, this is something for which automated analysis is ideally suited – far more than human analysis, in fact. Neither machine nor human can predict the future but one major benefit of automated analysis is the volume and granularity of time series data that it generates, which enable very sophisticated predictive models to be developed. Of course, humans still need to interpret the results of these models but they provide a very robust basis for forward-looking decision-making that, again, is pretty well impossible with human analysis. Last but not least the question of competitive advantage where, again, automated analysis offers huge benefits. The reason is that automated systems can analyse a hundred companies just as easily and only a few seconds longer than it takes to analyse one. The breadth and depth of competitive information this creates is huge and much, much greater than is practical – or cost-effective – using human analysis.

Like Reply Report

This is a great debate. It's something we think about quite a lot at Networked Insights. Our R&D team has some thoughts on our blog (http://bit.ly/gRb8T) that I think you'll find relevant. Thanks again for the great debate, Alex Fortney

Like Reply Report

Let me comment the subject from the engineering point of view. It seems to follow from the great discussion above that there are three main requirements as to the monitoring online media: real-time quickness, accuracy and understanding the meaning of the discourse monitored. As I see, we all agree, that the third needs specific human abilities and involvement in the monitoring process. So, the problem is how to construct human-operated Monitor accurate enough and fast enough. What does it mean "accurate enough"? What level of accuracy we need? Some experts seem to overestimate the weight of accuracy, and compete to attain it as close as possible to 100%. However, accuracy is costly. It costs both work and time. 100% accuracy is nonsense from the economic and technical point of view. Problem of optimum accuracy is similar to the old problem of quality assurance in manufacturing sector. Industrial statisticians made great progress when they replaced 100% quality control with the Statistical Process Control (SPC), based on the study of only the small fraction of pieces (samples). Some experts of sentiment analysis of online media texts urge to collect 100% of utterances of positive or negative opinions in the texts examined (sorry for simplification) in order to attain the highest possible accuracy. They apply strong computer applications to do that and attain the accuracy of 70-90%. This is not necessary. It is not so hard to attain 97% accuracy of the sentiment's changes measurement (on 0,95 confidence level) using SPC approach, provided that the results of sophisticated discourse analysis (human work!) are implemented in the Monitor. In this way the Monitor can be built as human-operated and computer-aided tool and process for real-time monitoring.

Like Reply Report