The birth of sentiment analysis

x Sponsored content on Research Live and in Impact magazine is editorially independent.
Find out more about advertising and sponsorship.

After World War II ended, it was widely assumed that Winston Churchill would be elected Prime Minister. He had led the nation through the war and Britain had won. All the media expected him to win the General Election in 1945, and the polls confirmed this. So the landslide victory of the Labour Party was a shock. This was the first major failure of ‘scientific’ polling to predict an election result. The only research organisation to forecast that the Labour Party would win was Mass Observation.

At the beginning of 1945, Gallup put the Labour share of the vote 20% behind that of the Conservatives. The polls continued to show a significant Conservative lead throughout the campaign. The actual result of the election in July was a Labour win with a 7% lead over the Conservatives. On the day of the election, Gallup published a poll that was very close to the actual result, but it was too late to have any influence. Mass Observation had predicted that Labour would win well in advance. At the time, this forecast was discounted because it was not based on a ‘scientific sample’. Mass Observation had synthesised information from a mixture of self-selected samples and qualitative work that included listening to conversations in public places. It had developed a way of classifying the formation of public opinion, which involved six stages:

1. What a person says to a stranger  (e.g. an interviewer with a questionnaire)
2. What a person says to an acquaintance
3. What a person says to a friend
4. What a person says to a partner or lover
5. What a person writes in a personal diary
6. What a person says to themselves or dreams

Mass observation argued that traditional quantitative polling elicited opinions at level 1. In the case of the 1945 election, no-one wanted to criticise Churchill publicly; however, conversations with friends were much more open, as people tried to find out if others were also thinking that Britain needed a change of leader. Mass Observation believed that opinions not yet fully formed were inaccessible to traditional surveys until the person was close to voting. Diaries and overheard conversations among friends were likely to give a much better indication of the issues and the way undecided voters, or those influenced by social norms, were likely to vote. 

Social media analysis and mobile ethnography are now accepted research techniques. In 1986, Mollie Tarrant, the research director at Mass Observation in its early years, expressed frustration at the narrow and superficial focus of quantitative research at that time. She wished she had been able to use video diaries; she would have been delighted to have had access to the automated text and image-coding software that is available now. What the Mass Observers were hoping to create 70 years ago is now possible at an affordable cost. They were the first to develop the ideas that underlie sentiment analysis. 

Adam Phillips is a Fellow of MRS and managing director of Real Research