Research in the dock

Based on the various emails and calls we received after the article was published, it’s clearly a priority within the industry, but what about the fundamental building blocks of the research process? Do questionnaires, and the subsequent interpretation of the data they provide us with, face up to the same level of scrutiny?

The Sun recently published a story headlined ‘1 in 5 Brit Muslims’ sympathy for jihadis: ‘Wake-up call’ after Paris blitz’. The story claimed that “nearly one in five British Muslims has some sympathy with those who have fled the UK to fight for IS in Syria.” The Independent Press Standards Organisation (IPSO) has since stated more than 2,600 people – a record number of complainants – had made contact with them over the front page piece, which was based on a survey by polling organisation Survation. The methods by which research agencies collect and analyse data were under the spotlight once again.

Following the widespread backlash, Survation quickly dismissed any involvement in The Sun’s interpretation of the study’s data, but were they wrong to take on such a controversial study?

Interestingly, YouGov (who usually poll for The Sun) declined the opportunity to run the survey. The company later stated: “To survey Britain’s Muslim population, particularly at a time of such heightened sensitivities, requires the kind of time, care, and therefore cost, that is beyond a newspaper’s budget.”

Survation had previously conducted a similar study on behalf of Sky News earlier this year, which presumably was the reason it was approached by The Sun. In March 2015, the company polled 1,000 Muslims and 1,000 non-Muslims about various issues including British values, terrorism, discrimination, integration, the influence of the UK security services and radicalisation.

what’s in a question?

The controversial ‘sympathy’ question was asked in the earlier Sky News poll, with the following figures observed:

"Which of the following statements is closest to your view?" 

• I have a lot of sympathy with young Muslims who leave the UK to join fighters in Syria (Muslims: 7.8%; Non-Muslims: 4.3%)
• I have some sympathy with young Muslims who leave the UK to join fighters in Syria (Muslims: 20.1%; Non-Muslims: 9.4%)
• I have no sympathy with young Muslims who leave the UK to join fighters in Syria (Muslims: 61.0%; Non-Muslims: 76.8%)

Sky News led its analysis of this initial study’s results with the headline “Majority Have No Sympathy With Extremists”. In the seventh paragraph of the article, it was stated “Sympathy with those leaving the UK to fight for or marry terrorist groups in Syria was highest among women. However, a majority of Muslims and non-Muslims said they had no sympathy for those joining extremist groups.”

If this study had garnered as much publicity as The Sun’s subsequent survey, this particular question and interpretation of data surely would have been criticised. The ambiguity of ‘joining fighters in Syria’ should have been avoided. Were they talking about those who leave the country to join ISIS, or those on the opposing side? Sky News had obviously taken it to mean the former in the interpretation of the data (“fight for or marry terrorist groups”).

Building on this, what is the definition of sympathy? Pity? Sadness? Understanding or supporting? In a subsequent press release, Survation discussed its own interpretation of the word:

The term “Sympathy” is understood by most people to mean the understanding of the distress or need of another person. Empathy and sympathy are often used interchangeably – empathy meaning the ability to share someone else’s feelings. Both these terms are very different to, and not to be confused with, agreeing with or supporting those feelings, and this is key to understanding the meaning of our polling.”

If we, as market researchers, are determining what words are understood to mean by respondents without actually asking them directly, we are setting a very dangerous precedent.

two for the price of one?

Another of the Sky News questions was a little on the confusing side:

“Which of the following statements is closest to your view?”:

• British Muslims are doing enough to integrate into British society
• British Muslims are not doing enough to integrate into British society
• It is not important for British Muslims to integrate into British society

Fairly early on in our research careers, we are taught that a question should address only one variable at a time. Asking if British Muslims are/are not doing enough to integrate, whilst simultaneously asking if it has any importance is lazy question writing. To most, this would be a two part question.

head to head

Throughout the Sky News study, a range of statements are pitted against each other, in a ‘head to head’ style. Are the values of British society compatible with the values of Islam, or are they not? Is it the responsibility of Muslims to condemn terrorist acts carried out in the name of Islam, or is it not? Are the actions of the police and MI5 contributing to the radicalisation of Muslims, or are they not? There is no scope whatsoever for strength of feeling to be established. You are either one or the other – no inbetween, no light and shade. A ‘don’t know’ response has been provided, but nothing can be assumed from that, other than the respondent genuinely doesn’t know which answer to pick or their opinion hasn’t been catered for. This kind of ‘A vs B’ style of polling is perfect for today’s 24 hour news world, where all you need is a simple statistic to kickstart hours and hours of debate and conjecture, but should it be tolerated by professional market researchers?

Clearly, this survey should have been dissected and condemned when it was published, but Sky’s uncontroversial headline – that the majority have no sympathy with extremists – ensured it went, to some extent, under the radar. Just another news soundbite.

There are obviously many other issues relating to this survey, in terms of the way it was recruited, conducted and analysed. When the research industry and the media lock horns again, we should hope that the researchers should be more involved in setting the agenda. How much responsibility do we really have over the findings we hand over to our clients? Should we demand to be involved with press releases and news stories when we have collected the data in question? The most important thing is that we retain our integrity – ultimately we should not shy away from communicating concerns over sensitive projects for the sake of turning a quick buck.

Laura Finnemore is a senior research executive at McCallum Layton