Scotland’s big question

I’ve worked in public opinion polling in the past, and indeed have polled on Scottish independence before, so a few people have asked me what I think about the question that has just been released by the Scottish National Party for a referendum on Scottish independence:

“Do you agree that Scotland should be an independent country? (Yes/No)”

I’ll go out on a limb here and say it’s not the worst. But it’s not great either.

Let’s start with a positive point. SNP leader Alex Salmond has said the question is “short, straightforward and clear” and he’s right, it is. I think anybody would be able to understand it. As we are talking about a referendum, it is necessary to plan for every adult in the country to be able to read and understand the question and I think that has been achieved. The question seems to comply with the following element of the Market Research Society’s code of conduct:

“Members must take reasonable steps to ensure that the design and content of the data collection process or instrument is appropriate for the audience being researched”

But as a member of the MRS I wouldn’t write this question myself and use it in a questionnaire. The main reason is that it is not balanced. The MRS code of conduct states that:

“Members must take reasonable steps to ensure that respondents are not led towards a particular point of view”

By asking us whether we agree that Scotland should be an independent country, the question leads us towards the ‘agree’ response. The way it is framed assumes a particular value position. It is saying ‘Do you agree (with us) that Scotland should be an independent country? Eliminating bias is questionnaire design 101, and not doing so is really not acceptable. Questions should be written so they are balanced and assume no value position, for example:

‘Do you agree or disagree that Scotland should be an independent country?’

‘Should Scotland be an independent country?’

But let’s consider the content of the question and intended use of the results. Thinking about what the SNP want to achieve, this is a good question to drum up a bit of national pride. They ask us whether Scotland should be an independent country, a wording which has the potential to elicit an idealistic agree response from anyone with any leaning towards patriotism (and there are plenty of them in Scotland) particularly when the ballot form is tarted up with national imagery such as saltires (sorry, I mean the Scottish Government logo). We are not asked to judge when independence should happen, or whether it will work. We are just asked to say if, in an ideal world, it should be the case. And that’s a legitimate question.

There’s a caveat to that though. We all know that a ‘yes’ vote will lead to independence, so those in Scotland who do agree that it should be an independent country but don’t agree that it will work in practice will be forced to lie with a no vote. So, given the potential use of the result, it is unclear whether the question meets the following MRS quality standard:

“Members must take reasonable steps to ensure that respondents are able to provide information in a way that reflects the view they want to express”

Linked to this, when the SNP get the results they will take the percentage yes vote and if it is high enough they will use it to start the process of independence. But is that appropriate based on the question? Not really. The answer to this question is not hugely practical, it just tells you what the nation thinks might be good if we lived in a vacuum. Don’t forget this is only one of a big pile of subtly different questions that the SNP could have asked. I bet if they asked ‘Should Scotland become an independent nation next year?’ or ‘Could Scotland function as an independent nation?’ they’d get an entirely different response. It is therefore unclear whether the question meets the following MRS quality standard:

“Members must take reasonable steps to ensure that responses are capable of being interpreted in an unambiguous way”

Here, by the way, is the original referendum question that the SNP were proposing in 2007, which is much more practical and balanced:

“The Scottish Parliament has decided to consult people in Scotland on the Scottish Government’s proposal to negotiate with the Government of the United Kingdom to achieve independence for Scotland:

Put a cross (X) in the appropriate box

I AGREE that the Scottish Government should negotiate a settlement with the Government of the United Kingdom so that Scotland becomes an independent state.

I DO NOT AGREE that the Scottish Government should negotiate a settlement with the Government of the United Kingdom so that Scotland becomes an independent state.”

So is the new question the right question? Ach, well that depends on what you are trying to achieve. Would I have asked it that way as a professional researcher? No. Is the proposed approach ethical? Questionable. But if you are trying to drum up support for Scottish independence it is well framed, so good on the SNP – I have a certain respect for their cunning. And if the no vote wins, that implies a no to all the other more ethical less idealistic questions too. This is them giving it their best shot.

Or is it? This alternative referendum question that I sourced from Facebook seems much more appropriate than the official version.

Ruth Stevenson runs Ruthless Research. This article first appeared on her blog