Reflecting reality: The future of ‘nat rep’

x Sponsored content on Research Live and in Impact magazine is editorially independent.
Find out more about advertising and sponsorship.

We all think we know what ‘nationally representative’ means. It is a sample that broadly reflects the population as a whole, taking into account gender, age, region and occupation, and occasionally social class. It is a tried and tested method that has been in use in research for decades, and is the backbone of much of the industry’s work with the public.

The question, however, is whether this relatively narrow concept of nationally representative samples is reflective of the modern UK. Since the first census took place in 1801, there have been radical changes across society, and the UK is arguably a vastly different country from what it was only a few decades ago.

Some have argued that these societal changes necessitate expanding national representative samples, to include factors such as race, gender preference, disability and sexuality. “You are marginalising a subset of people if you are not including their voices in a research piece,” says Graham Idehen, director, customer success – Europe, Middle East and Africa at Lucid, and a co-founder of the Colour of Research (CoRe). “Therefore, it is not ‘nat rep’ if you’re not including them.”

He adds that the problem is one of definition. “It’s always going to come back to the same thing – what is ‘nat rep’? What does this mean to you? How are you defining it? How are you communicating that to your audience?”

Inclusive data efforts

Work to define ‘nat rep’ is ongoing, both inside and outside of the market research industry. The Office for National Statistics (ONS) has been one of the most active in exploring how to take factors such as race and disability into better account through national representative studies.

In September 2021, the ONS published the findings of its inclusive data taskforce, which included recommendations that data producers adopt an intersectional approach to exploring and presenting equalities data, to avoid potentially misleading single-characteristic analyses.

The ONS taskforce also highlighted some of the data gaps that exist within the generally accepted nationally representative methodology: non-residential populations, such as those in care homes, prisons or who are homeless; harder-to-reach groups, such as the travelling community; religion; income; children; non-internet users; and sexuality. Barriers to participation in research were also noted, including a lack of trust in the authorities and in government statistics, or the burden of repeated requests for participation.

The Market Research Society (MRS) also launched a steering group in 2021 (see boxout) to examine how nationally representative samples could become more inclusive. Rebecca Cole, managing director at Cobalt Sky, is leading the steering group, and says that, historically, many social surveys in the UK focused on social class ahead of other characteristics, such as race, which have been the focus of nationally representative surveys in countries including the US. This could mean, Cole argues, that we might not be getting a truly representative sample.

“One of the main drivers coming out at the moment, in a lot of client demand, is they want to know that the sample they are buying is representative in terms of ethnicity,” she says. “Unless we shift – unless we first and foremost start asking these questions – we have no way of proving, of knowing, whether our sample is representative or not.”

Idehen sees green shoots emerging in the research industry’s desire to address issues around inclusion, but is concerned that many companies are scared to change from tried and tested practices. “There’s a healthy appetite for wanting to start evolving nat rep, and engaging in the discussion around nat rep and what it means,” he explains. “I think there’s hesitation in terms of actually implementing and executing it because of the fear of the unknown – ‘what does this mean I have to do? How does this impact my business? How does this impact my data?’”

Gender

Transgender, non-binary and gender-diverse people were spotlighted as a major data blindspot in the ONS report. Independent researcher Jo Shaw says that the concept of nationally representative sampling is a relic from an age when gender nonconformity was not well established in mainstream consciousness. “Nat rep comes from an era when society was used to being grouped into very large boxes,” she explains. “Quite often now, the way the world is means that’s not going to work in the same way.”

Shaw says that a focus purely on the numerical size of a group within society – such as the transgender population – has had a detrimental effect. “It is important their voices are heard,” she says. “But if you approach it just from a numerical, statistical standpoint, you could say they are only a small part of society, so they are not very important, mathematically, in the work you are doing. If you just crunch the numbers, there’s lots of important groups – important in that they have questions, need answers and need someone to listen to them – that can’t be heard. It is important that agencies come to the issue with an understanding and sensitivity to it.”

Part of the problem is segmentation, which Shaw says is used primarily to make life easier for the researcher, rather than for the benefit of society as a whole, or for minority groups. Shaw says that market research underplays the important impact it can have on helping achieve greater societal acceptance for marginalised groups.

“The research industry’s job is not just to record what is going on in society; it takes an important role in defining and shaping that society,” Shaw says. “Society uses the data we produce to understand itself. If we get it wrong, society will understand itself in a different way – maybe even the wrong way.”
Pounds and pence

Cost is one barrier to expanding nationally representative surveys. Currently, to commission surveys with minority audiences, panels tend to charge a premium for each survey. The lower the incidence of a sample, the higher the cost per interview (CPI).

In 2019, Michael Brown, chair of MRSpride, and group partner, insight and cross-culture, at UM, collated aggregated CPIs following a sample brief sent to eight UK panels to understand the cost issue. Brown found that the average CPI to research gay or bisexual men was £12 and for lesbian or bisexual women this rose to £13. In comparison, the CPI for sending a survey to an audience of ‘mums’ was £1.50.

Theo Francis, owner and managing director of GuineaPig, and CoRe co-founder, says that businesses should accept that clients want to have diverse samples, taking ethnicity into account, and swallow the costs. “I’ve started up my own fieldwork agency, but I don’t look at the costs that I have to incur to provide the services that clients are looking for and say, ‘well, the client should be paying me to be able to offer them’ – I have to do that as a business,” he argues. “If you want to compete in the future, you’ve got to do what you’ve got to do to compete, and this is just one of those things – it’s a cost of doing business.”

A question of wording

The lack of best-practice guidance on the specifics of wording is a problem, according to Cole. She fears that issues such as gender identity are not being included in samples because of reticence on the part of researchers over the best language to use, and the best categories to employ. For sensitive information pertaining to a person’s health, sexuality, identity or race, getting the language right is crucial if people are to be encouraged to engage with the industry and respond to surveys when requested, thereby providing a more representative sample.

“If you’re an agency and you want to ask about sexual orientation, there isn’t an MRS, or otherwise provided, best-practice guide that says ‘here’s the wording; we’ve consulted with groups; we’ve all agreed; here are the responses; here are the orders’,” Cole says. “Of course, that makes people anxious, because the last thing people want to do in this situation is ask something badly – and that’s actually a big barrier to agencies and clients doing it right now.” Cole is currently leading the MRS’s work to provide the necessary best-practice guidance (see boxout). Trixie Cartwright, head of Europe at Ipsos Interactive Services, says that wording for categories such as gender identity needs to be both clear and inclusive – ensuring everyone has an answer option that is relevant to them – while remaining understandable for people who may have little to no knowledge of what different categories might mean. For example, on gender identity, do you opt for more than 50 genders, like Facebook does, or have a single category for people who do not identify as a man or a woman?

Cartwright is reviewing the best practice on wording with the MRS, to provide some clarity. “When you are doing research, you want as many people to answer as honestly as possible, and you need them to be able to answer to a level that you would want to classify them at, and that gives them the ability to answer,” she explains. “You want to be inclusive and respectful of the community you want to include, but you need to be mindful that the question does not confuse or cause a reduced response rate from others.”

There are other aesthetic issues with which any guidelines on national representation will have to grapple. “There are some people who think that ethnicity should be multi-coded, for example – that people can identify with more than one ethnicity,” Cole says. “There’s lots of conversation around ordering, and whether we make it alphabetical. On the face of it, that sounds sensible – why should white be at the top? However, there’s been work done by the ONS that says that, if you do that, you can get an under-representation on mixed ethnicities. People tend to tick the first thing they see that they identify with, so if you alphabetise, ‘Asian’ and ‘black’ sees a bump, and mixed ‘white and Asian’, mixed ‘white and black’ goes down.”

There is also the question of whether nationally representative surveys are even a good way of getting the views of minority groups. Robert Agnew, managing director UK at Norstat, says that broad categories – such as black, Asian and minority ethnic (Bame) – encompass a wide variety of views and cultural norms, making such blanket terms unhelpful. “I think the term Bame is too ‘on the nose’, and I don’t see it as inclusive – I see it as the opposite,” he says. “If the objective is to share minorities’ voices, a public poll in ‘nat rep’ is not the best way to do it. It is designed to do the opposite – to give you a UK view, rather than a subset of that audience.”

Encouraging participation

Shaw says there is a trust issue facing research, and work is needed to encourage people from minority groups to take part. “It is difficult to get anyone to talk to you for the fear of being misrepresented,” she argues. “Job one is to restore trust.”

Perhaps the biggest elephant in the room is the lack of diversity in the industry itself. “My view is that, if you are looking for diverse thought and diverse analysis, and different ways of understanding data, there’s an argument for overrepresentation of people from minority backgrounds,” Agnew states. “If you have five people in a room who all read data in the same way, you only need one of them. The fact we are behind in representation has the potential to undermine our industry.”

This, he adds, would support recruiting people from minority groups into panels and qualitative research, as there would be more employees who have a better understanding of those groups and their relationship with the market research industry.

Shaw adds that seeing and meeting people who have shared experiences could be crucial. “It can make a lot of difference hearing from someone who you think ‘I recognise some of their experience, they might not stitch me up, as they have experienced this too’,” she adds. “There is a continuing recruitment issue, allowing people from minority groups across the country to come through into the industry – there are way too few.”