Survey sensitivity

Yet, when asked, nine in every 10 ( 88%) of us say sectarianism is still a problem for our country. As part of ScotCen’s on-going annual Scottish Social Attitudes survey, which interviews around 1,500 people each year, we sought to understand why sectarianism is still present in a society that otherwise tends to be considered fairly liberal in its attitudes.

The report, Public Attitudes to Sectarianism in Scotland, explores what people across the whole of Scotland think about the issue – something that has never been done previously, meaning it will serve as a crucial piece of evidence in the Scottish Government’s on-going commitment to tackle sectarianism. The survey addresses a number of complex issues such as perceptions of religious conflict in Scottish society, direct experience of sectarianism, views on what contributes to sectarianism and what could be done to address the issue.

Shared understanding

Designing the questions for this study was a challenging process. We had to ask ourselves: will people in Scotland have a shared understanding of what sectarianism means? And in an increasingly secular Scotland, would people think of themselves in terms of religious identity?
A technique that greatly aided the process of designing these questions and gave us confidence that people would understand what we were asking was cognitive interviewing: a practice qualitative in nature used to test and improve survey questions.

During a cognitive interview, the interviewer asks a survey question (as normal) and then follows this with a series of probes that aim to detect any problems with the questions, understanding of particular terms used, participants’ comfort level in answering questions and how appropriate the answer categories are.

We felt that people with different religious identities (or lack of them) may have different views about sectarianism. Since sectarianism in Scotland is historically rooted in tensions between Catholics and Protestants, it was important to explore whether people of Christian background often identified themselves as Catholic or Protestant. Through cognitive interviewing we learned more about the complex nature of people’s religious identities: we spoke to those with clear-cut religious identities, but also those who did not think of themselves as particularly religious but who still saw themselves as Catholic or Protestant.

Meanwhile some participants described themselves as Christian but did not identify with a particular denomination. This meant that asking just about belonging to a particular religion was not enough for the purpose of this survey; we needed to find ways to explore both people’s religion and their broader cultural religious identity.

And how about using the term sectarianism? To find out if this was a term the general public would understand we asked people what it meant to them. We also tested the phrase “tensions between Catholics and Protestants” to find out if this could work as an alternative. The results of cognitive testing showed that people recognised and understood the term sectarianism more clearly and consistently than its paraphrased version, which gave us confidence to use it in the survey.

Negative associations

Cognitive interviewing not only made it easier to decide on the appropriate language to use throughout the survey but it also helped us make decisions about the order of the questions. While we found that the term ‘sectarianism’ was widely understood, it also carried a lot of negative associations such as football-related violence and discriminatory attitudes. For this reason we decided to place some of the questions related to people’s views on discrimination against Catholics/Protestants and prejudice against the two groups at the start of the survey, before referring to the term sectarianism directly in the latter part.

Sometimes, however, you have to admit that not everything can be explored in a survey, as some concepts proved to be too context-dependent. For example, we wanted to capture some of the more subtle expressions of sectarianism, such as jokes about Protestants and Catholics, so during the cognitive interviews we posed a hypothetical situation in which a friend makes a joke about a Protestant. We found ourselves facing a flurry of clarification questions from the participants such as: was the joke offensive or funny? Was it directed at someone present at the room? Do people who are making these jokes know each other well? This type of issue might be better suited for a qualitative study which can explore more of the contextual nuances.

Overall, we succeeded in addressing many issues relevant to the debates on sectarianism, which bring religion, politics and football into a complex mix. Thanks to the information gained during the cognitive interview process we were able to design a range of questions that will better inform policy makers, particularly the Scottish Government who commissioned the study, on the views of the public on sectarianism.

Anna Marcinkiewicz is a researcher at ScotCen.