The first census in Britain was conducted in the Gaelic kingdom of Dalriada, on the west coast of Scotland, in the seventh century. Senchus fer n-Alban (The history of the men of Scotland) differed vastly from modern censuses in that its aim was to gather records for military and tax purposes, not to inform statistics about the nation.

Better known is the next effort to record the population – the Domesday Book, commissioned by William the Conqueror in 1086. Again, the survey was taken to determine what tax revenue the king could extract from his subjects.

More than 700 years later, after over-population fears were stoked when Thomas Malthus published his Essay on the Principle of Population, the first modern census for England and Wales was conducted in 1801. In the UK, the census has taken place every decade since, with only one exception during World War II.

Today, censuses do more than simply count the population. They offer the most detailed snapshot of a country at a given point in time, providing data that informs decisions and planning by local authorities and governments in fundamental areas such as housing and transport. They are also used by researchers and businesses to understand society and make decisions.

“They are vital,” says David Hussey, director of statistics at the National Centre for Social Research (NatCen). “In social research, it’s important for our surveys to get really accurate estimates for whatever we’re measuring, and census data is vital for the sampling and the weighting stages.”

Census data is particularly valuable for researchers looking at details beyond age and sex at local authority level and above, as it includes more detailed information. For NatCen’s face-to-face surveys, such as British Social Attitudes Survey and the Health Survey for England, it uses the census for stratification – to order a sample before it is drawn.

One example of how census data is used in sampling is when researchers are interested in bespoke areas in and around cities that are not necessarily defined by groups, local authorities or a region. “The only way we know what those cities look like, particularly in terms of age and gender, is to build up census data from its lowest level, which is output area level – typically 150 households,” Hussey explains.

Independent demographer Piers Elias agrees that the census is key, particularly when it comes to ethnicity data. “It’s the main source for ethnicity and – even though it’s only once every 10 years – there is no other source that covers everyone in that sort of detail. The levels of change also tend to stay in the same areas – an area of high ethnicity in 2011 will probably be the same area in 2021.”

Data on areas smaller than local authority level is crucial for councils allocating funding, and making decisions as wide-ranging as where to supply public health workers to planning bus routes, adds Elias.

Frances Darlington-Pollock, lecturer in population geography at the University of Liverpool, uses census data in her work to understand inequalities and the differences in migration patterns. “Particularly with Brexit, and all this uncertainty, we know there are differences in the way in which people move house,” she says. “Some move because of a favourable change in their circumstances; others remain trapped in an area they would rather leave. Using census data and linking information about the deprivation of an area allows you to understand how people move.

“That’s crucial for planning – it matters for where you put schools and what types of health services you need. Without census data, you can’t do that.”

Navigational issues

One of the obvious limitations of a decennial census is its frequency.

“Census data is a little inflexible because, by the time you get to 2019 – when the previous census was in 2011 – that’s quite a long time ago. Sometimes, you just have to bite the bullet and say ‘this is the best information we’ve got’,” says Hussey, who adds that ethnicity data is one specific example of this.

“An improvement would be if there was more up-to-date information available, but, of course, the trade-off is in the quality and accuracy versus the timeliness – and there’s always going to be a trade-off there.”

For Darlington-Pollock, it’s not the 10-year gap that causes issues, but rather the lag in publishing the different types of data. While doing her PhD, she needed the Sample of Anonymised Records (SAR) – “census microdata” that gives the full breadth of census questions for a small percentage of the population. “That wasn’t released until the middle to end of 2014, so you’re waiting quite a long time – because it is such a high standard, they have to make sure it’s confidential and secure,” she says.

“That delay is a problem. It’s something I know the Office for National Statistics (ONS) is working to address, but it’s a huge dataset, so it’s a big undertaking.”

The microdata is also useful for market researchers, says Corrine Moy, global vice-president of marketing science at GfK. “The SAR is a great source of data for understanding person-level relationships and doing more in-depth analysis.”

One census blind spot is how much people earn, because it is deemed too sensitive to ask about, although the ONS is looking at supplying the information from administrative data. Income would be a “really useful addition” for the commercial research world, says Moy. “To profile individuals, we often use the SAR. We would typically use this dataset to obtain profiles of certain populations – for example, mothers with kids under five. The ability to include income in this profiling would be valuable,” she adds.

In the context of the United Kingdom, the ‘census’ isn’t a single survey; devolved nations carry out their own censuses. So, although it happens on the same day, it involves three surveys run by the ONS in England and Wales, the National Records of Scotland (NRS) and the Northern Ireland Statistics and Research Agency (Nisra).

“The national statistician has responsibility for providing the UK-wide population estimates and we undertake that through a body called the UK census committee, where we meet with the head of the NRS and head of Nisra,” says Iain Bell, deputy national statistician for population and public policy at the ONS, and the senior officer responsible for the census transformation programme.

“We meet every quarter to make sure we’re moving forward together and harmonising where we can, while still respecting the needs of different parts of the country.”

Hard questions

Devolution can pose challenges for researchers, however. “It does make working with census data a bit challenging,” says Hussey. “It just takes longer, and the same thing is called something different; there is no such thing as a ‘lower super output area’ in Scotland – it’s called a ‘data zone’.”

With such national significance, the census affects everybody, so it’s not surprising that it attracts its fair share of public interest – and occasional controversy – over what is being asked. Gender identity is the current area of interest. While exact wording is yet to be determined, the 2021 census in England and Wales, and the Scottish census will ask about gender identity and sexual orientation for the first time.

In the US, the 2020 census was at the centre of a political row after the Trump administration declared it would ask respondents if they are US citizens, provoking criticism from human rights groups and businesses. The question was dropped after a Supreme Court ruling, but the President said the government would collect the data using information already held. The issue around asking about citizenship was that it would negatively affect turnout, highlighting a key issue: censuses must find the right balance between accuracy and response rate – particularly given their longevity and influence.

In the UK, the census undergoes rigorous consultation and testing to ensure questions are asked sensitively and only if necessary. The ONS looks at the topics covered – determined by the areas of highest user need – as well as how questions are worded.

Censuses are costly affairs that require large amounts of resources. So, in recent years, many countries have begun examining how to source information without relying solely on the census, looking to administrative data already held by governments. In some cases, this is replacing decennial surveys altogether.

Sweden, for example, no longer conducts a census, but holds data on its population register, while – in France – a partial census is carried out annually, and the results published as averages over five years. Not only do alternative census approaches lessen the burden on citizens, they could also give more up-to-date information.

This need to adapt is the focus of the ONS’s census transformation programme; 220 years after the first census, the 2021 survey could be the last of its kind.

“As part of the wider Census Data Collection and Transformation Programme, using data-sharing provisions in the Data Economy Act, ONS is exploring how administrative data could be used to replace a traditional decennial census after 2021,” says Bell.

“What the census does is really two things: the first is to give us the population estimates in detail, down to geographic level; and the second is to give us a description of society today, through all the other questions, such as equalities, housing, travel to work, types of work, health. Our research programme is looking at both aspects.”

In June 2019, the ONS published an update on its work on the new system of population and migration estimates, looking at the administrative data it has from government departments and how well it could estimate the population from that. It was largely successful in reducing the over-coverage seen in previous research. “We know how to deal with under-coverage – and the level of under-coverage is similar, if not slightly better, for some ages than the level of the raw census data,” says Bell.

But, he adds: “The big task is still where would we get all the information for all the other characteristics in the census? We’ve got much more research to do there.”

The aim of the programme is to publish a recommendation in 2023 on the future of the census. The National Records of Scotland (NRS), which is responsible for Scotland’s census, has also been exploring the future of the country’s socio-demographic statistics.

“The census provides incredibly detailed and valuable information, but it does only happen once every 10 years and involves huge effort,” says Amy Wilson, head of Scotland’s Census 2021.

“For 2021, we concluded that the best way to do this is to carry out a country-wide data collection, but we will be using some administrative statistics to help us plan and process the census.

“Going forward, we will continue to look at whether our current model is the best – and most efficient – way to deliver high-quality data, or whether there are different ways in which we could meet those needs.”

Direction of travel

Using government data for information about the population is no simple task, however. “There are all sorts of problems with data sources that aren’t designed to count people – they are designed to administer a particular set of people,” says Elias: “The worry with not having a census is that you’ll have no idea, really, when things are going in the wrong direction.”

Darlington-Pollock adds that it’s important for researchers to remember the value of the census: “Even when we’ve got all these amazing new forms of data, the census is there and offers something additional to that. You definitely get a split between people who are traditional census users and people who are more on the side of big data, but I think many people see the value of both.”

The census maps out the people of the country, acting as a baseline grid that serves multiple purposes, but this is really just the starting point. As Bell says, the value of the census is in how it is used. “The estimate of the population and describing the social condition of the country is not an end in itself – it’s about the information that it gives people in local authorities and central government, and to  employers wanting to know if they’re reflecting the diversity of the local area in their workplace. The importance is really in those decisions.”

Learning from Africa

Censuses in Africa face unique challenges and offer insights for statistics experts elsewhere. We spoke to Sophie Elfar and Garnett Compton from the ONS international development team to find out more.

“The census is probably more important in Africa because there aren’t a lot of other data sources available,” says Compton, who works at UNECA’s African Centre for Statistics in Addis Ababa. “It not only provides the characteristics of your population and housing, but covers a lot of other topics – such as births, because there isn’t a standard civil registration system. It does more than western censuses.”

In Ghana, Elfar, strategic adviser to the Ghana Statistical Service, says census data is particularly important for sharing evidence with development partners. “Without good data, there’s a risk that marginalised groups become invisible and are not included in national development programmes. Without good data, they don’t have a voice.”

For its next census in 2020, Ghana, like several other African countries, is using handheld tablets for the first time. “We’re working with international development partners and other national statistics institutes to see if we can use their tablets and return them after our own census,” says Elfar. “That’s a mammoth task.”

UK census teams can learn a lot from Africa, says Compton. “The UK census plans, developments and methods are shared with countries that do similar censuses. You don’t get a completely fresh look at your census, whereas – when you try to do things in Africa – it gives you the opportunity to say ‘maybe we should look at that again’, or to clarify why you are doing what you’re doing.”

This article was first published in Impact’s October 2019 issue.