Six rebel ideas to make sampling an innovation priority

Sampling’s press centres on automation and quality. Automation reduces sampling to a commodity item, and quality issues are worrying. Yet its lack of innovation (apart from automation) is more concerning.

It’s concerning because it suggests we’ve forgotten what research is about. People. Not platforms. Not blockchain. Not dashboards. People.

And because sampling is about which people we speak to, we should be making it an innovation priority, not – as sampling risks being – an automated commodity.

Time to rebel

You may be thinking, ‘how do we innovate sampling?’ I was too. Then I read Rebel Ideas: The Power of Diverse Thinking by Matthew Syed.

The book argues that problems are best solved by combining different ways of thinking instead of using one way of thinking. The concept of cognitive diversity isn't only fascinating, but we can use it to innovate sampling.

1. Contravene convention about countries

Insight briefs typically state that research should focus on a specified set of countries, the normal response to which is a proverbial head nod. Rebel Ideas suggests there’s a better way to approach this –  ignore geographical borders.

For example, next time you're designing UK product test research, why not include sample from Finland, too? Yes, Finland. Or any country that isn’t the UK for that matter. Why? Because UK culture informs how Brits think. And Finnish culture informs how Finns think.

This means Finnish people will ignore UK cultural norms and view the product you’re researching with a Finnish cultural lens. This different view will provide you with a greater breadth of feedback than if you only used a UK sample.

Not convinced? Then consider this. Immigrants or their children founded 45% of Fortune 500 companies (New American Fortune 500 Report, 2019 ). This means 45% of Fortune 500 companies founders’ thinking wasn't dictated by US cultural norms, allowing founders of these companies to think differently to American natives. And differentiation – as we know – is everything.

2. Argue against averages. Encourage the extremities

Research often needs the ‘average’ person’s perspective (despite Gilbert Daniels showing that the concept of an ‘average’ person is a dangerous one).

Usability and UX research can benefit by sampling people at the extremes. People who are as far from ‘average’ as possible – e.g. people who have low tech-savviness or attention disorders.

But why?

Because if someone at the extremes can use a UX solution, the masses will likely be able to use it too. Nintendo’s game designer Shigeru Miyamoto used this thinking to create the Nintendo-Wii. Wii’s over 110m sales confirmed Miyamoto’s approach was effective.

3. Demand difference. Shun sameness

Often our samples specify that people in them have the same attitude. But, speaking to people who all think the same risks limiting what samples can achieve.

Syed calls this ‘collective blindness’. This blindness caused the CIA’s failure to spot clues about 9/11. The CIA created this blindness by using the same criteria to recruit all their analysts, which meant they all thought the same. 

Resultantly, they didn’t spot clues about the 9/11 attacks. Clues that people with a basic knowledge of Islamic culture would’ve done. The CIA’s standardised recruitment methods meant they had no analysts with such knowledge.

But what does this mean for sampling? If samples have people who think differently, this will limit a sample’s collective blindness.

In doing so it increases the sample’s frame of reference – how much of a problem it can view. A cognitively diverse sample will solve a problem better than one with shared attitudes.

4. Get nosey about people’s networks

We include people in samples based on who they are and what they do. But the people who individuals in samples know (their network) are as important, because  people with bigger networks have access to more ways of thinking, meaning they’re more likely to be diverse thinkers themselves.

Research showing that the top 20% performing executives have diverse networks evidences this. People with large networks are more than good leaders; they're also valuable research participants.

5. Ignore expert individuals. Embrace expert groups

We speak to experts to understand well-informed opinions – and this expertise is valuable. However, individual experts with expertise in highly specific areas have limited value to us, because experts are so focused on their own area of expertise, they have a limited breadth of thinking. Philip Tetlock says that this causes experts to ‘intellectually blind themselves’.

Ensuring our sample’s aggregate expert opinion with expertise from different areas can negate intellectual blindness. This is because, as James Surowiecki emphasises in his book The Wisdom of Crowds, aggregating expertise cancels out both errors and individual’s intellectual blindness.

6. Be a rebel with a cause

Syed compellingly shows cognitive diversity’s power, but he also makes it clear that there’s more to it than bundling different people together.

Instead, if you want to design a rebellious – cognitively diverse – sample, consider: what out-of-scope countries are relevant to a brief? What scale of extremities should we use? How different do samples need to be? What kind of networks should individuals have? What aggregated expertise will best cancel out individual errors?

Is rebellion revolution or evolution?

Rebel ideas can help us innovate for the future, but ironically, rebellion and revolution already have a place in sampling’s history courtesy of Pierre Simon Laplace. Laplace first used sampling to estimate France’s population in 1786 and favoured evolution over revolution. This means he may not approve of the term ‘rebel idea’. But I’m sure he’d make an exception in this instance because rebel ideas truly can help sampling evolve.

Jack Miles is senior research director at Northstar Research