This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Find out more here

OPINION1 February 2012

Scotland’s big question

Opinion

As the Scottish National Party plans a referendum on Scottish independence, Ruth Stevenson of Ruthless Research wonders whether they’re asking the right question.

I’ve worked in public opinion polling in the past, and indeed have polled on Scottish independence before, so a few people have asked me what I think about the question that has just been released by the Scottish National Party for a referendum on Scottish independence:

“Do you agree that Scotland should be an independent country? (Yes/No)

I’ll go out on a limb here and say it’s not the worst. But it’s not great either.

“As a member of the MRS I wouldn’t write this question myself and use it in a questionnaire. The main reason is that it is not balanced”

Let’s start with a positive point. SNP leader Alex Salmond has said the question is “short, straightforward and clear” and he’s right, it is. I think anybody would be able to understand it. As we are talking about a referendum, it is necessary to plan for every adult in the country to be able to read and understand the question and I think that has been achieved. The question seems to comply with the following element of the Market Research Society’s code of conduct:

“Members must take reasonable steps to ensure that the design and content of the data collection process or instrument is appropriate for the audience being researched”

But as a member of the MRS I wouldn’t write this question myself and use it in a questionnaire. The main reason is that it is not balanced. The MRS code of conduct states that:

“Members must take reasonable steps to ensure that respondents are not led towards a particular point of view”

By asking us whether we agree that Scotland should be an independent country, the question leads us towards the ‘agree’ response. The way it is framed assumes a particular value position. It is saying ‘Do you agree (with us) that Scotland should be an independent country? Eliminating bias is questionnaire design 101, and not doing so is really not acceptable. Questions should be written so they are balanced and assume no value position, for example:

‘Do you agree or disagree that Scotland should be an independent country?’

‘Should Scotland be an independent country?’

But let’s consider the content of the question and intended use of the results. Thinking about what the SNP want to achieve, this is a good question to drum up a bit of national pride. They ask us whether Scotland should be an independent country, a wording which has the potential to elicit an idealistic agree response from anyone with any leaning towards patriotism (and there are plenty of them in Scotland) particularly when the ballot form is tarted up with national imagery such as saltires (sorry, I mean the Scottish Government logo). We are not asked to judge when independence should happen, or whether it will work. We are just asked to say if, in an ideal world, it should be the case. And that’s a legitimate question.

There’s a caveat to that though. We all know that a ‘yes’ vote will lead to independence, so those in Scotland who do agree that it should be an independent country but don’t agree that it will work in practice will be forced to lie with a no vote. So, given the potential use of the result, it is unclear whether the question meets the following MRS quality standard:

“Members must take reasonable steps to ensure that respondents are able to provide information in a way that reflects the view they want to express”

Linked to this, when the SNP get the results they will take the percentage yes vote and if it is high enough they will use it to start the process of independence. But is that appropriate based on the question? Not really. The answer to this question is not hugely practical, it just tells you what the nation thinks might be good if we lived in a vacuum. Don’t forget this is only one of a big pile of subtly different questions that the SNP could have asked. I bet if they asked ‘Should Scotland become an independent nation next year?’ or ‘Could Scotland function as an independent nation?’ they’d get an entirely different response. It is therefore unclear whether the question meets the following MRS quality standard:

“Members must take reasonable steps to ensure that responses are capable of being interpreted in an unambiguous way”

Here, by the way, is the original referendum question that the SNP were proposing in 2007, which is much more practical and balanced:

“The Scottish Parliament has decided to consult people in Scotland on the Scottish Government’s proposal to negotiate with the Government of the United Kingdom to achieve independence for Scotland:

Put a cross (X) in the appropriate box

I AGREE that the Scottish Government should negotiate a settlement with the Government of the United Kingdom so that Scotland becomes an independent state.

I DO NOT AGREE that the Scottish Government should negotiate a settlement with the Government of the United Kingdom so that Scotland becomes an independent state.

So is the new question the right question? Ach, well that depends on what you are trying to achieve. Would I have asked it that way as a professional researcher? No. Is the proposed approach ethical? Questionable. But if you are trying to drum up support for Scottish independence it is well framed, so good on the SNP – I have a certain respect for their cunning. And if the no vote wins, that implies a no to all the other more ethical less idealistic questions too. This is them giving it their best shot.

Or is it? This alternative referendum question that I sourced from Facebook seems much more appropriate than the official version.

Ruth Stevenson runs Ruthless Research. This article first appeared on her blog

2 Comments

6 years ago

Ruth, the alternative referendum one is brilliant!

Like Report

6 years ago

There is another aspect to consider. In legal matters "agreement" is legally binding. it is a irrevocable commitment. The SNP question elicits a clear Yes or No to a commitment to proceed to independence. It imparts the clear message that this requires the voter to agree or disagree. Commit, one way or the other. In contrast, asking "should..." is open to the charge of being ambiguous, eg. a) Yes Scotland "should" where the 'should is taken as only a possibility (But not yet/ not really/ when we are ready.etc. etc) b) No Scotland should not (but I support independence as it is inevitable) and other variations and alternatives. So the "Do you agree" bit removes ambiguity, requires commitment and gives a clear measure of the level of those committed as a basis ofr proceeding - or heaven forfend, Not.

Like Report