Recent controversies have the industry debating the ethics of polling and the detail of the British Polling Council’s rules. These debates turn on disclosure and client transparency, which of course are important issues.

But a book I read recently got me thinking about the ethics of my trade more broadly, and whether polling can mislead even when rules like the BPC’s are followed to the letter.

The book was David W. Moore’s The Opinion Makers. Moore, a former senior Gallup editor, argues that pollsters don't just measure public opinion, they manufacture it.

After years polling the public, my instinct is to defend polling. But even though Moore’s book was published nearly 20 years ago, it felt particularly timely, and left me with an uneasy question: am I an opinion manufacturer, too?

How polls manufacture opinion

Moore argues that even with balanced, well-designed questions, many polls manufacture opinion. They press people on issues they neither know nor care about, so pollsters misrepresent a view that was never really there.

He uses polling before the Iraq war as an example. Around six weeks before the invasion, Gallup published a poll suggesting broad public backing for sending troops to remove Saddam Hussein. Support looked strong: 59% in favour, 38% opposed, just 3% with no opinion.

But Moore and his colleagues showed how fragile support was by asking how strongly people felt. Supporters were asked whether they'd be upset if the government didn't invade; opponents, whether they'd be upset if it did.

The picture shifted dramatically. Only 29% were firmly in favour and 30% firmly opposed, while 41% either didn't mind either way or had no opinion at all.

Using examples like this, Moore splits opinion into two types: ‘directive’ strongly held views, and ‘permissive’ – offered when asked, but lightly held. By measuring these permissive views and reporting them as considered, pollsters drift from measuring to manufacturing.

A manufactured finding can have all sorts of downstream effects. It misleads people, it can shape the very opinion it claimed to measure, and hands a policy’s supporters a sense of a mandate that was never there.

Soul (or poll) searching

I can think of questions I've asked that would fall foul of the same critique. A while back I ran a poll asking whether people supported an annual land value tax to replace stamp duty and council tax. Around two in five backed it, while one in five opposed.

But had many of those 1,500 people given a land value tax a moment’s thought before we asked? Very few, I'd imagine.

It again highlights the caution required before treating the answers as anything more than surface-level proxy responses.

What we can do about it

We can all do more to design and report polls with greater care. For questions measuring voters’ views on policy or issues, I'd suggest a four-part framework: SUBS (sorry. I couldn't resist making an acronym).

The framework draws on principles most good pollsters already know and regularly practice. But we're probably all guilty of letting it slip at times. The components are:

1. Salience. Find out whether people care about the issue you're asking about.

People will back a policy in the abstract even when the underlying issue is one they never think about or care for. If it barely registers, that tells you how much weight to give the subsequent numbers.

2. Uncertainty: Quantify and underline uncertainty.

Ideally that means measuring how much people know or have heard about the issue you're asking about. When measuring support, offer both a "neither" and a "don't know", so people can show ambivalence, indifference or uncertainty, and avoid doing so only to rebase your percentages to make them disappear altogether.

3. Balance. Without it, you heighten the risk of a manufactured result.

We all know the importances of unbiased, balanced questions. Yet how often do agree/disagree statements creep back into our questionnaires? These formats compound the problem. If respondents already tend to opine on issues they don't have a settled view on, a design that encourages biases like acquiescence or social desirability only makes it worse.

4. Strength. Avoid formats that conflate strong and weak opinions.

Moore’s follow-up question exploring strength of feeling was revealing, but there isn't space for it on every survey. However, even a simple scale does useful work: items like ‘somewhat support’ or ‘somewhat oppose’ help draw out more permissive perspectives.

More measuring, less manufacturing

The four SUBS points are partly about good questionnaire design. But the framework works best applied more widely, running through analysis, reporting and commentary, too.

My point about measuring strength is a good example. Many pollsters, me included, already run a five-point scale for strength of support, so the suggestion is far from groundbreaking. But we then merge the outer two points into each side to get a summary figure, and that’s usually the only number reported in any meaningful way.

A SUBS mindset means holding on to the difference between somewhat and strongly supporting something. Part of the job, then, is working with clients to draw out nuances like that, or at least keep them in view, when results are written up.

I'm conscious, however, that this won't fix everything. It won't always stop a headline writer overstating or exaggerating what a poll really found.

But if we routinely apply SUBS-like thinking in our own analysis, reporting and commentary, we can help those consuming polls read them better, too. That way, we help people see the opinions that are really there and do less to manufacture what isn't.

Robert Struthers is head of public perceptions at BMG Research